2021-03-09T08:32:59.077 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-09T08:32:59.081 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949083 --name kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/lockdep} --owner scheduled_kchai@teuthology --verbose -- /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949083/orig.config.yaml 2021-03-09T08:32:59.149 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-09T08:32:59.218 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949083 branch: wip-kefu-testing-2021-03-09-1428 description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/lockdep} email: tchaikov@gmail.com first_in_suite: false job_id: '5949083' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: '8.3' overrides: admin_socket: branch: wip-kefu-testing-2021-03-09-1428 ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 3 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: lz4 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug monc: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-kefu-testing-2021-03-09-1428 sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 owner: scheduled_kchai@teuthology priority: 60 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 sleep_before_teardown: 0 suite: rados suite_branch: wip-kefu-testing-2021-03-09-1428 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 targets: smithi099.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDg1b1HraNq0AWyoPGdbrFvsAebk5Cfx+u2LSwiCX81xITsIQImbZN3B1l7yxhIaVH7fSKirZjbIYmwJyr8NxjRex05vaa59nNTD4LNJBh+Q71nKGnf/AEp8MlXkMwdlQWSWIIodcuSI+TD6pVi4vnYfpqeJBa11cisUnPeHLk8I9FiI4byUVR561j/Zvbx1h0QlzbthtnL8Hai3NLWRMpwjVmVdAeRPaeMJGIz5oo4ho8YDo5y2I16u0NwVLTsd+OYasmj/tyx3oyy7BV8V8Aw1opyWEG8eZ6lhhXKVtw/Q1IulfN/NMh+MgUkaDm2hmJdriHj5k4cpBurxXEqXbBrpgvvoiCEOVGQwzAM//dR6jZOwRkMIiWt3IZ9O1bq/NSs86HPEK/iEvqTntw0UpcNuybBLRT+JgN3aizF6i8HsBMPOT8tVz9qFea+1koIOgvWoy9gtAe5N5A1xDupjl00r79Nt01kczSMxCRYJnMU2W39MIQRedsgjGOj0sLWU5k= smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6vWtGsGfQ4Z1POo2m1ZI+yBkT5G9TWJiO6WZt4CYAtyc9y2Mt3Nr4jp7dRiQNJke0xAGppqPvol+5C9vn2Tdm2vTrpzgustdXvwwJaOxxQnPL4oJEJLE3e2EUjTRFHJVdcMc/NUQm08FOLgV4HZTP6wM3n7nl4n1VEZaM4Dl3Ndd78TH9288qVZiEbRYPqJzHj+iYKQEgZN6RluEthreT7XYBP/djZVIDqXS4uxVVLL/IeyT100Rzq8/BlE89zMJbH0sW5PeS78pygHQQN/bICycJEDzkWwnKVCqIiQYruD5+E8Af1pcrfxWuXElsfN4IMe234Ia/vtzuazB5WEWACsNkuPIYu81pSvqNpFlIeYXGWcB/dlIHZwYPW+iybTpYQcvkRnutkBvNl296MToqwIcap/pMbhyL4fSJnCa87tZovjwAJQOZk3kkUzOobqbXISqKlAOTretXIdT//dcqvDcqoPLfEnNxJoaiukgV4plHDVMvAK1LVzKBjW+x9+M= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology_branch: master teuthology_sha1: 3082387bba74fcd24c9700593d10418152d53c97 timestamp: 2021-03-09_08:22:40 tube: smithi user: kchai verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.25973 2021-03-09T08:32:59.219 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa; will attempt to use it 2021-03-09T08:32:59.231 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa/tasks 2021-03-09T08:32:59.232 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-09T08:32:59.234 INFO:teuthology.task.internal:Checking packages... 2021-03-09T08:32:59.323 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '89fb622b10ca2e58fe0c913a8956377a886e2ab4' 2021-03-09T08:32:59.323 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-09T08:32:59.326 INFO:teuthology.packaging:ref: None 2021-03-09T08:32:59.327 INFO:teuthology.packaging:tag: None 2021-03-09T08:32:59.327 INFO:teuthology.packaging:branch: wip-kefu-testing-2021-03-09-1428 2021-03-09T08:32:59.328 INFO:teuthology.packaging:sha1: 89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:32:59.329 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-kefu-testing-2021-03-09-1428 2021-03-09T08:32:59.492 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-1742.g89fb622b 2021-03-09T08:32:59.492 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-09T08:32:59.502 INFO:teuthology.task.internal:no buildpackages task found 2021-03-09T08:32:59.502 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-09T08:32:59.523 INFO:teuthology.task.internal:Saving configuration 2021-03-09T08:32:59.540 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-09T08:32:59.549 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-09T08:32:59.566 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 08:28:01.561680', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949083', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzUg94byRTJWCP63Kt2pTz4bfpaVMfGt1CU+X1TCIEQRaDvUczi2eLtAYmFrxPx0VWDkTdE7vjECtkHHhC9JjM=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi099.front.sepia.ceph.com'} 2021-03-09T08:32:59.583 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-09 08:28:01.561662', 'description': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949083', 'locked_by': 'scheduled_kchai@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPLZv4AS3ADZV/FcPuF+ZeYeU1iLw/PMi8BwOq+uCIKebS/K0IlnZalmjKi2C5+ejqdM90wrsovsySF6KPWjyfQ=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi195.front.sepia.ceph.com'} 2021-03-09T08:32:59.583 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-09T08:32:59.592 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-03-09T08:32:59.593 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-03-09T08:32:59.593 INFO:teuthology.run_tasks:Running task console_log... 2021-03-09T08:32:59.665 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5835e0a950>, signals=[15]) 2021-03-09T08:32:59.666 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-09T08:32:59.675 INFO:teuthology.task.internal:Opening connections... 2021-03-09T08:32:59.675 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2021-03-09T08:32:59.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:32:59.746 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2021-03-09T08:32:59.747 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:32:59.818 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-09T08:32:59.828 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2021-03-09T08:32:59.869 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2021-03-09T08:32:59.870 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2021-03-09T08:32:59.923 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Linux" 2021-03-09T08:32:59.924 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="8" 2021-03-09T08:32:59.924 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2021-03-09T08:32:59.924 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2021-03-09T08:32:59.925 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="8" 2021-03-09T08:32:59.925 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el8" 2021-03-09T08:32:59.925 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Linux 8" 2021-03-09T08:32:59.926 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2021-03-09T08:32:59.926 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-09T08:32:59.926 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2021-03-09T08:32:59.926 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-09T08:32:59.927 INFO:teuthology.orchestra.run.smithi099.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-09T08:32:59.927 INFO:teuthology.orchestra.run.smithi099.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-09T08:32:59.928 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2021-03-09T08:32:59.945 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2021-03-09T08:32:59.960 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2021-03-09T08:32:59.961 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2021-03-09T08:33:00.015 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Linux" 2021-03-09T08:33:00.015 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8" 2021-03-09T08:33:00.015 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2021-03-09T08:33:00.016 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2021-03-09T08:33:00.016 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8" 2021-03-09T08:33:00.016 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2021-03-09T08:33:00.016 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Linux 8" 2021-03-09T08:33:00.017 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2021-03-09T08:33:00.017 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-09T08:33:00.017 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2021-03-09T08:33:00.017 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-09T08:33:00.018 INFO:teuthology.orchestra.run.smithi195.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-09T08:33:00.018 INFO:teuthology.orchestra.run.smithi195.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-09T08:33:00.018 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2021-03-09T08:33:00.034 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-09T08:33:00.051 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-09T08:33:00.066 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-09T08:33:00.067 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2021-03-09T08:33:00.071 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2021-03-09T08:33:00.085 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-09T08:33:00.096 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-09T08:33:00.096 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2021-03-09T08:33:00.125 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2021-03-09T08:33:00.149 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-09T08:33:00.198 INFO:teuthology.run_tasks:Running task kernel... 2021-03-09T08:33:00.216 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}} 2021-03-09T08:33:00.217 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-09T08:33:00.217 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2021-03-09T08:33:00.232 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-240.10.1.el8_3.x86_64 2021-03-09T08:33:00.233 INFO:teuthology.task.kernel:Running kernel on smithi099: 4.18.0-240.10.1.el8_3.x86_64 2021-03-09T08:33:00.233 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2021-03-09T08:33:03.312 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - AppStream 20 MB/s | 6.3 MB 00:00 2021-03-09T08:33:04.441 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - Base 17 MB/s | 2.3 MB 00:00 2021-03-09T08:33:04.904 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - Extras 101 kB/s | 9.2 kB 00:00 2021-03-09T08:33:06.065 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Linux 8 - AppStream 5.6 MB/s | 6.3 MB 00:01 2021-03-09T08:35:00.985 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Linux 8 - BaseOS 20 kB/s | 2.3 MB 01:53 2021-03-09T08:35:01.863 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Linux 8 - Extras 19 kB/s | 9.2 kB 00:00 2021-03-09T08:35:02.446 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 17 MB/s | 8.9 MB 00:00 2021-03-09T08:35:04.326 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2021-03-09T08:35:05.787 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-240.10.1.el8_3.x86_64 is already installed. 2021-03-09T08:35:05.787 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-03-09T08:35:06.001 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:35:06.016 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:35:06.017 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:35:06.017 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:35:06.017 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:35:06.017 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 4.3 M 2021-03-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2021-03-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 30 M 2021-03-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 26 M 2021-03-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:35:06.018 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:35:06.019 INFO:teuthology.orchestra.run.smithi099.stdout:Install 3 Packages 2021-03-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 60 M 2021-03-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 83 M 2021-03-09T08:35:06.020 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:35:06.180 INFO:teuthology.orchestra.run.smithi099.stdout:(1/3): kernel-4.18.0-240.15.1.el8_3.x86_64.rpm 27 MB/s | 4.3 MB 00:00 2021-03-09T08:35:07.166 INFO:teuthology.orchestra.run.smithi099.stdout:(2/3): kernel-modules-4.18.0-240.15.1.el8_3.x86 22 MB/s | 26 MB 00:01 2021-03-09T08:35:07.420 INFO:teuthology.orchestra.run.smithi099.stdout:(3/3): kernel-core-4.18.0-240.15.1.el8_3.x86_64 22 MB/s | 30 MB 00:01 2021-03-09T08:35:07.424 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:35:07.424 INFO:teuthology.orchestra.run.smithi099.stdout:Total 43 MB/s | 60 MB 00:01 2021-03-09T08:35:07.509 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:35:07.834 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:35:07.834 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:35:08.960 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:35:08.979 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:35:11.290 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:35:11.778 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-core-4.18.0-240.15.1.el8_3.x86_64 1/3 2021-03-09T08:35:14.847 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-4.18.0-240.15.1.el8_3.x86_64 1/3 2021-03-09T08:35:15.152 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-4.18.0-240.15.1.el8_3.x86_64 2/3 2021-03-09T08:35:18.027 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-4.18.0-240.15.1.el8_3.x86_64 2/3 2021-03-09T08:35:18.236 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-4.18.0-240.15.1.el8_3.x86_64 3/3 2021-03-09T08:35:38.261 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-4.18.0-240.15.1.el8_3.x86_64 3/3 2021-03-09T08:35:38.524 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-4.18.0-240.15.1.el8_3.x86_64 3/3 2021-03-09T08:35:38.524 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-4.18.0-240.15.1.el8_3.x86_64 1/3 2021-03-09T08:35:38.525 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-core-4.18.0-240.15.1.el8_3.x86_64 2/3 2021-03-09T08:35:38.809 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-4.18.0-240.15.1.el8_3.x86_64 3/3 2021-03-09T08:35:38.810 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:35:39.211 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:35:39.211 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:35:39.211 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:35:39.212 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core-4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:35:39.212 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:35:39.212 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:35:39.212 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:35:39.281 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel --last | head -n 1 2021-03-09T08:35:39.407 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-240.15.1.el8_3.x86_64 Tue 09 Mar 2021 08:35:18 AM UTC 2021-03-09T08:35:39.408 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:35:39.409 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-240.10.1.el8_3.x86_64 Expected: 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:35:39.409 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2021-03-09T08:35:39.427 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-240.10.1.el8_3.x86_64 2021-03-09T08:35:39.428 INFO:teuthology.task.kernel:Running kernel on smithi195: 4.18.0-240.10.1.el8_3.x86_64 2021-03-09T08:35:39.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2021-03-09T08:35:42.517 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-8 - AppStream 27 MB/s | 6.3 MB 00:00 2021-03-09T08:35:43.764 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-8 - Base 9.1 MB/s | 2.3 MB 00:00 2021-03-09T08:35:44.215 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-8 - Extras 129 kB/s | 9.2 kB 00:00 2021-03-09T08:35:44.788 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Linux 8 - AppStream 12 MB/s | 6.3 MB 00:00 2021-03-09T08:36:51.855 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Linux 8 - BaseOS 35 kB/s | 2.3 MB 01:06 2021-03-09T08:36:52.727 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Linux 8 - Extras 19 kB/s | 9.2 kB 00:00 2021-03-09T08:36:53.387 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 15 MB/s | 8.9 MB 00:00 2021-03-09T08:36:55.313 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 5.1 MB/s | 24 kB 00:00 2021-03-09T08:36:56.891 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-240.10.1.el8_3.x86_64 is already installed. 2021-03-09T08:36:56.892 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-03-09T08:36:57.104 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:36:57.119 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:36:57.120 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:36:57.120 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:36:57.120 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:36:57.121 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 4.3 M 2021-03-09T08:36:57.121 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2021-03-09T08:36:57.121 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 30 M 2021-03-09T08:36:57.121 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 26 M 2021-03-09T08:36:57.121 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:36:57.122 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:36:57.122 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:36:57.122 INFO:teuthology.orchestra.run.smithi195.stdout:Install 3 Packages 2021-03-09T08:36:57.122 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:36:57.123 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 60 M 2021-03-09T08:36:57.123 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 83 M 2021-03-09T08:36:57.123 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:36:57.346 INFO:teuthology.orchestra.run.smithi195.stdout:(1/3): kernel-4.18.0-240.15.1.el8_3.x86_64.rpm 19 MB/s | 4.3 MB 00:00 2021-03-09T08:36:58.115 INFO:teuthology.orchestra.run.smithi195.stdout:(2/3): kernel-modules-4.18.0-240.15.1.el8_3.x86 26 MB/s | 26 MB 00:00 2021-03-09T08:36:58.496 INFO:teuthology.orchestra.run.smithi195.stdout:(3/3): kernel-core-4.18.0-240.15.1.el8_3.x86_64 22 MB/s | 30 MB 00:01 2021-03-09T08:36:58.499 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:36:58.500 INFO:teuthology.orchestra.run.smithi195.stdout:Total 44 MB/s | 60 MB 00:01 2021-03-09T08:36:58.586 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:36:58.923 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:36:58.923 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:37:00.050 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:37:00.062 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:37:02.324 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:37:02.813 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-4.18.0-240.15.1.el8_3.x86_64 1/3 2021-03-09T08:37:05.884 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-4.18.0-240.15.1.el8_3.x86_64 1/3 2021-03-09T08:37:06.254 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-4.18.0-240.15.1.el8_3.x86_64 2/3 2021-03-09T08:37:09.134 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-4.18.0-240.15.1.el8_3.x86_64 2/3 2021-03-09T08:37:09.314 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-4.18.0-240.15.1.el8_3.x86_64 3/3 2021-03-09T08:37:29.244 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-4.18.0-240.15.1.el8_3.x86_64 3/3 2021-03-09T08:37:29.512 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-4.18.0-240.15.1.el8_3.x86_64 3/3 2021-03-09T08:37:29.512 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-4.18.0-240.15.1.el8_3.x86_64 1/3 2021-03-09T08:37:29.513 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-4.18.0-240.15.1.el8_3.x86_64 2/3 2021-03-09T08:37:29.789 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-4.18.0-240.15.1.el8_3.x86_64 3/3 2021-03-09T08:37:29.790 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:37:30.154 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:37:30.154 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:37:30.155 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:30.155 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core-4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:30.155 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:30.155 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:37:30.156 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:37:30.225 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel --last | head -n 1 2021-03-09T08:37:30.351 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-240.15.1.el8_3.x86_64 Tue 09 Mar 2021 08:37:09 AM UTC 2021-03-09T08:37:30.353 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:30.353 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-240.10.1.el8_3.x86_64 Expected: 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:30.353 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-03-09T08:37:30.354 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-03-09T08:37:30.354 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-03-09T08:37:30.354 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-03-09T08:37:30.355 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi099.front.sepia.ceph.com, path=None, version=distro) 2021-03-09T08:37:30.355 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel --last | head -n 1 2021-03-09T08:37:30.471 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-4.18.0-240.15.1.el8_3.x86_64 Tue 09 Mar 2021 08:35:18 AM UTC 2021-03-09T08:37:30.473 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:30.473 DEBUG:teuthology.orchestra.run.smithi099:> sudo rpm -qi grub2-tools 2021-03-09T08:37:30.504 INFO:teuthology.orchestra.run.smithi099.stdout:Name : grub2-tools 2021-03-09T08:37:30.505 INFO:teuthology.orchestra.run.smithi099.stdout:Epoch : 1 2021-03-09T08:37:30.505 INFO:teuthology.orchestra.run.smithi099.stdout:Version : 2.02 2021-03-09T08:37:30.505 INFO:teuthology.orchestra.run.smithi099.stdout:Release : 90.el8 2021-03-09T08:37:30.505 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: x86_64 2021-03-09T08:37:30.505 INFO:teuthology.orchestra.run.smithi099.stdout:Install Date: Mon 01 Mar 2021 06:17:56 PM UTC 2021-03-09T08:37:30.506 INFO:teuthology.orchestra.run.smithi099.stdout:Group : System Environment/Base 2021-03-09T08:37:30.506 INFO:teuthology.orchestra.run.smithi099.stdout:Size : 9197207 2021-03-09T08:37:30.506 INFO:teuthology.orchestra.run.smithi099.stdout:License : GPLv3+ 2021-03-09T08:37:30.506 INFO:teuthology.orchestra.run.smithi099.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-03-09T08:37:30.507 INFO:teuthology.orchestra.run.smithi099.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-03-09T08:37:30.507 INFO:teuthology.orchestra.run.smithi099.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-03-09T08:37:30.507 INFO:teuthology.orchestra.run.smithi099.stdout:Build Host : kbuilder.bsys.centos.org 2021-03-09T08:37:30.507 INFO:teuthology.orchestra.run.smithi099.stdout:Relocations : (not relocatable) 2021-03-09T08:37:30.507 INFO:teuthology.orchestra.run.smithi099.stdout:Packager : CentOS BuildSystem 2021-03-09T08:37:30.508 INFO:teuthology.orchestra.run.smithi099.stdout:Vendor : CentOS 2021-03-09T08:37:30.508 INFO:teuthology.orchestra.run.smithi099.stdout:URL : http://www.gnu.org/software/grub/ 2021-03-09T08:37:30.508 INFO:teuthology.orchestra.run.smithi099.stdout:Summary : Support tools for GRUB. 2021-03-09T08:37:30.508 INFO:teuthology.orchestra.run.smithi099.stdout:Description : 2021-03-09T08:37:30.508 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:37:30.509 INFO:teuthology.orchestra.run.smithi099.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-03-09T08:37:30.509 INFO:teuthology.orchestra.run.smithi099.stdout:customizable bootloader with modular architecture. It supports a rich 2021-03-09T08:37:30.509 INFO:teuthology.orchestra.run.smithi099.stdout:variety of kernel formats, file systems, computer architectures and 2021-03-09T08:37:30.509 INFO:teuthology.orchestra.run.smithi099.stdout:hardware devices. 2021-03-09T08:37:30.509 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:37:30.510 INFO:teuthology.orchestra.run.smithi099.stdout:This subpackage provides tools for support of all platforms. 2021-03-09T08:37:30.511 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-03-09T08:37:30.511 INFO:teuthology.task.kernel:Updating grub on smithi099 to boot 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:30.511 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-03-09T08:37:31.246 INFO:teuthology.orchestra.run.smithi099.stderr:Generating grub configuration file ... 2021-03-09T08:37:33.311 INFO:teuthology.orchestra.run.smithi099.stderr:done 2021-03-09T08:37:33.313 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2021-03-09T08:37:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.QLxVKdMEpF 2021-03-09T08:37:33.364 DEBUG:teuthology.orchestra.run.smithi099:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QLxVKdMEpF 2021-03-09T08:37:33.394 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.QLxVKdMEpF 2021-03-09T08:37:33.554 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.QLxVKdMEpF is 6KB 2021-03-09T08:37:33.604 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.QLxVKdMEpF 2021-03-09T08:37:33.621 DEBUG:teuthology.orchestra.run.smithi099:> sudo /bin/ls /boot/loader/entries || true 2021-03-09T08:37:33.680 INFO:teuthology.orchestra.run.smithi099.stdout:631d84db90f7409fbdd390573ae89718-0-rescue.conf 2021-03-09T08:37:33.681 INFO:teuthology.orchestra.run.smithi099.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.10.1.el8_3.x86_64.conf 2021-03-09T08:37:33.681 INFO:teuthology.orchestra.run.smithi099.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.15.1.el8_3.x86_64.conf 2021-03-09T08:37:33.681 INFO:teuthology.orchestra.run.smithi099.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.el8.x86_64.conf 2021-03-09T08:37:33.683 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-set-default 631d84db90f7409fbdd390573ae89718-4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:33.822 DEBUG:teuthology.orchestra.run.smithi099:> sudo shutdown -r now 2021-03-09T08:37:33.825 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2021-03-09T08:37:33.825 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2021-03-09T08:37:33.825 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel --last | head -n 1 2021-03-09T08:37:33.940 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-4.18.0-240.15.1.el8_3.x86_64 Tue 09 Mar 2021 08:37:09 AM UTC 2021-03-09T08:37:33.943 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:33.943 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2021-03-09T08:37:33.974 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2021-03-09T08:37:33.975 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2021-03-09T08:37:33.975 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.02 2021-03-09T08:37:33.975 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 90.el8 2021-03-09T08:37:33.975 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2021-03-09T08:37:33.975 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Mon 01 Mar 2021 06:17:56 PM UTC 2021-03-09T08:37:33.976 INFO:teuthology.orchestra.run.smithi195.stdout:Group : System Environment/Base 2021-03-09T08:37:33.976 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 9197207 2021-03-09T08:37:33.976 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2021-03-09T08:37:33.976 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-03-09T08:37:33.977 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-03-09T08:37:33.977 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-03-09T08:37:33.977 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : kbuilder.bsys.centos.org 2021-03-09T08:37:33.977 INFO:teuthology.orchestra.run.smithi195.stdout:Relocations : (not relocatable) 2021-03-09T08:37:33.977 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : CentOS BuildSystem 2021-03-09T08:37:33.978 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2021-03-09T08:37:33.978 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2021-03-09T08:37:33.978 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2021-03-09T08:37:33.978 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2021-03-09T08:37:33.979 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:37:33.987 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-03-09T08:37:33.988 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2021-03-09T08:37:33.988 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2021-03-09T08:37:33.988 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2021-03-09T08:37:33.988 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:37:33.989 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2021-03-09T08:37:33.990 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-03-09T08:37:33.990 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:33.990 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-03-09T08:37:35.465 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2021-03-09T08:37:37.509 INFO:teuthology.orchestra.run.smithi195.stderr:done 2021-03-09T08:37:37.582 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2021-03-09T08:37:37.642 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.POQkccVkda 2021-03-09T08:37:37.642 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.POQkccVkda 2021-03-09T08:37:37.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.POQkccVkda 2021-03-09T08:37:37.844 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.POQkccVkda is 6KB 2021-03-09T08:37:37.883 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.POQkccVkda 2021-03-09T08:37:37.904 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2021-03-09T08:37:37.972 INFO:teuthology.orchestra.run.smithi195.stdout:631d84db90f7409fbdd390573ae89718-0-rescue.conf 2021-03-09T08:37:37.973 INFO:teuthology.orchestra.run.smithi195.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.10.1.el8_3.x86_64.conf 2021-03-09T08:37:37.973 INFO:teuthology.orchestra.run.smithi195.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.15.1.el8_3.x86_64.conf 2021-03-09T08:37:37.974 INFO:teuthology.orchestra.run.smithi195.stdout:631d84db90f7409fbdd390573ae89718-4.18.0-240.el8.x86_64.conf 2021-03-09T08:37:37.976 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default 631d84db90f7409fbdd390573ae89718-4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:37:38.119 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2021-03-09T08:38:08.158 INFO:teuthology.misc:Re-opening connections... 2021-03-09T08:38:08.159 INFO:teuthology.misc:trying to connect to ubuntu@smithi099.front.sepia.ceph.com 2021-03-09T08:38:08.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:38:08.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:38:26.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2021-03-09T08:38:26.500 INFO:teuthology.misc:trying to connect to ubuntu@smithi195.front.sepia.ceph.com 2021-03-09T08:38:26.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:38:26.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:39:26.562 DEBUG:teuthology.orchestra.remote:timed out 2021-03-09T08:39:26.572 DEBUG:teuthology.misc:waited 78.41250777244568 2021-03-09T08:39:27.574 INFO:teuthology.misc:trying to connect to ubuntu@smithi099.front.sepia.ceph.com 2021-03-09T08:39:27.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:39:27.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:39:27.874 DEBUG:teuthology.orchestra.run.smithi099:> true 2021-03-09T08:39:28.183 DEBUG:teuthology.misc:waited 80.02372717857361 2021-03-09T08:39:29.185 INFO:teuthology.misc:trying to connect to ubuntu@smithi195.front.sepia.ceph.com 2021-03-09T08:39:29.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:39:29.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:39:29.436 DEBUG:teuthology.orchestra.run.smithi195:> true 2021-03-09T08:39:29.744 DEBUG:teuthology.misc:waited 81.58464026451111 2021-03-09T08:39:30.745 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-03-09T08:39:30.745 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-240.15.1.el8_3.x86_64"... 2021-03-09T08:39:30.746 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2021-03-09T08:39:30.763 INFO:teuthology.orchestra.run.smithi099.stdout:4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:39:30.764 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.15.1.el8_3.x86_64 vs 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:39:30.764 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-03-09T08:39:30.764 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-03-09T08:39:30.765 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2021-03-09T08:39:30.765 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-240.15.1.el8_3.x86_64"... 2021-03-09T08:39:30.765 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2021-03-09T08:39:30.782 INFO:teuthology.orchestra.run.smithi195.stdout:4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:39:30.783 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.15.1.el8_3.x86_64 vs 4.18.0-240.15.1.el8_3.x86_64 2021-03-09T08:39:30.783 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-03-09T08:39:30.783 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-03-09T08:39:31.784 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-03-09T08:39:32.683 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T08:39:32.760 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2021-03-09T08:39:32.777 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-03-09T08:39:32.777 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-09T08:39:32.852 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2021-03-09T08:39:32.872 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-09T08:39:32.890 INFO:teuthology.task.internal:Creating test directory... 2021-03-09T08:39:32.893 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T08:39:32.902 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-09T08:39:32.921 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-09T08:39:33.001 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-09T08:39:33.017 INFO:teuthology.task.internal:Creating archive directory... 2021-03-09T08:39:33.017 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T08:39:33.024 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-09T08:39:33.167 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-09T08:39:33.179 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-09T08:39:33.180 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T08:39:33.187 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-09T08:39:33.214 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:39:33.216 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:39:33.226 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:39:33.227 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-09T08:39:33.229 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-09T08:39:33.260 INFO:teuthology.task.internal:Configuring sudo... 2021-03-09T08:39:33.261 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T08:39:33.270 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-09T08:39:33.297 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-09T08:39:33.428 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-09T08:39:33.429 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T08:39:33.432 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-09T08:39:33.447 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T08:39:33.523 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T08:39:33.601 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T08:39:33.635 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T08:39:33.684 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:39:33.685 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T08:39:33.749 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T08:39:33.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-09T08:39:33.851 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T08:39:33.928 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-09T08:39:33.976 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:39:33.977 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-09T08:39:34.042 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2021-03-09T08:39:34.045 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2021-03-09T08:39:34.111 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-03-09T08:39:34.146 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-03-09T08:39:34.578 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-09T08:39:34.589 INFO:teuthology.task.internal:Starting timer... 2021-03-09T08:39:34.589 INFO:teuthology.run_tasks:Running task pcp... 2021-03-09T08:39:34.624 INFO:teuthology.run_tasks:Running task selinux... 2021-03-09T08:39:34.640 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-03-09T08:39:34.641 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2021-03-09T08:39:34.720 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: [ OK ] 2021-03-09T08:39:34.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2021-03-09T08:39:34.786 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: [ OK ] 2021-03-09T08:39:34.788 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-09T08:39:34.788 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2021-03-09T08:39:34.814 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2021-03-09T08:39:34.814 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2021-03-09T08:39:34.867 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2021-03-09T08:39:34.868 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi099.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2021-03-09T08:39:34.869 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-03-09T08:39:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T08:39:34.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-03-09T08:39:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-09T08:39:34.921 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-09T08:39:34.922 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2021-03-09T08:39:34.963 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2021-03-09T08:39:34.987 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-09T08:39:35.000 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-03-09T08:39:35.000 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-09T08:39:35.017 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-09T08:39:35.020 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-09T08:39:35.733 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-09T08:39:37.494 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-09T08:39:39.857 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-09T08:39:39.931 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:39 +0000 (0:00:00.111) 0:00:00.111 ********* 2021-03-09T08:39:39.954 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:39 +0000 (0:00:00.022) 0:00:00.134 ********* 2021-03-09T08:39:41.601 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi195.front.sepia.ceph.com] 2021-03-09T08:39:41.652 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:41 +0000 (0:00:01.698) 0:00:01.832 ********* 2021-03-09T08:39:41.973 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-09T08:39:41.975 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:39:41.995 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:41 +0000 (0:00:00.343) 0:00:02.175 ********* 2021-03-09T08:39:42.590 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-03-09T08:39:42.591 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-09T08:39:42.616 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:42 +0000 (0:00:00.621) 0:00:02.796 ********* 2021-03-09T08:39:42.647 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-09T08:39:42.649 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] 2021-03-09T08:39:42.705 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:42 +0000 (0:00:00.088) 0:00:02.885 ********* 2021-03-09T08:39:42.711 INFO:teuthology.task.ansible.out: 2021-03-09T08:39:42.738 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:43.715 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:42 +0000 (0:00:00.058) 0:00:02.943 ********* TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} Tuesday 09 March 2021 08:39:42 +0000 (0:00:00.180) 0:00:03.124 ********* TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi195.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Tuesday 09 March 2021 08:39:43 +0000 (0:00:00.293) 0:00:03.418 ********* TASK [ansible-managed : Create the ansible user.] ****************************** ok: [smithi099.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Tuesday 09 March 2021 08:39:43 +0000 (0:00:00.060) 0:00:03.478 ********* TASK [ansible-managed : Delete the ansible users password.] ******************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:39:43 +0000 (0:00:00.140) 0:00:03.619 ********* TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi099.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Tuesday 09 March 2021 08:39:43 +0000 (0:00:00.243) 0:00:03.862 ********* 2021-03-09T08:39:44.571 INFO:teuthology.task.ansible.out: 2021-03-09T08:39:44.573 INFO:teuthology.task.ansible.out:TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279183.7502134-5704-70576067962980/source", "state": "file", "uid": 0} changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279183.3281846-5654-25109864502694/source", "state": "file", "uid": 0} 2021-03-09T08:39:44.594 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:44 +0000 (0:00:00.912) 0:00:04.774 ********* 2021-03-09T08:39:44.616 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:44 +0000 (0:00:00.021) 0:00:04.796 ********* 2021-03-09T08:39:45.186 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-03-09T08:39:45.188 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-09T08:39:45.189 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T08:39:45.231 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-09T08:39:45.233 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-09T08:39:45.299 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-09T08:39:45.303 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-09T08:39:45.469 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:45 +0000 (0:00:00.853) 0:00:05.649 ********* 2021-03-09T08:39:45.489 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:45 +0000 (0:00:00.019) 0:00:05.669 ********* 2021-03-09T08:39:46.095 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-09T08:39:46.096 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] 2021-03-09T08:39:46.148 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-03-09T08:39:46.150 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:46 +0000 (0:00:00.659) 0:00:06.329 ********* 2021-03-09T08:39:46.893 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-09T08:39:46.895 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.165109", "end": "2021-03-09 08:39:46.846979", "rc": 0, "start": "2021-03-09 08:39:46.681870", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-09T08:39:46.927 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:46 +0000 (0:00:00.778) 0:00:07.107 ********* 2021-03-09T08:39:47.114 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-09T08:39:47.117 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] 2021-03-09T08:39:47.156 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-09T08:39:47.158 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009667", "end": "2021-03-09 08:39:47.121614", "failed_when_result": false, "rc": 0, "start": "2021-03-09 08:39:47.111947", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T08:39:47.179 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:47 +0000 (0:00:00.252) 0:00:07.359 ********* 2021-03-09T08:39:47.211 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:47 +0000 (0:00:00.031) 0:00:07.391 ********* 2021-03-09T08:39:47.252 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-09T08:39:47.254 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:47.278 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:47 +0000 (0:00:00.066) 0:00:07.458 ********* 2021-03-09T08:39:47.280 INFO:teuthology.task.ansible.out: 2021-03-09T08:39:47.374 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:47.401 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:47 +0000 (0:00:00.122) 0:00:07.581 ********* 2021-03-09T08:39:47.501 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-09T08:39:47.503 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:47.526 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:47 +0000 (0:00:00.125) 0:00:07.706 ********* 2021-03-09T08:39:47.621 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-09T08:39:47.623 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:47.655 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:47 +0000 (0:00:00.129) 0:00:07.836 ********* 2021-03-09T08:39:47.688 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-09T08:39:47.690 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.170867", "end": "2021-03-09 08:39:47.654038", "rc": 0, "start": "2021-03-09 08:39:47.483171", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-09T08:39:47.712 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:47 +0000 (0:00:00.056) 0:00:07.893 ********* 2021-03-09T08:39:47.756 INFO:teuthology.task.ansible.out: 2021-03-09T08:39:47.758 INFO:teuthology.task.ansible.out:TASK [Filter the managed_admin_users list] ************************************* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:47.782 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:47 +0000 (0:00:00.069) 0:00:07.962 ********* 2021-03-09T08:39:47.948 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-09T08:39:47.950 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.021691", "end": "2021-03-09 08:39:47.923413", "failed_when_result": false, "rc": 0, "start": "2021-03-09 08:39:47.901722", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-09T08:39:47.971 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:47 +0000 (0:00:00.189) 0:00:08.151 ********* 2021-03-09T08:39:48.042 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-03-09T08:39:48.044 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:39:48.065 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:48 +0000 (0:00:00.094) 0:00:08.246 ********* 2021-03-09T08:39:48.083 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-09T08:39:48.085 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:48.108 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:48 +0000 (0:00:00.042) 0:00:08.288 ********* 2021-03-09T08:39:48.197 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-03-09T08:39:48.199 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:48.219 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:48 +0000 (0:00:00.111) 0:00:08.400 ********* 2021-03-09T08:39:48.300 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:48.321 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:48 +0000 (0:00:00.101) 0:00:08.502 ********* 2021-03-09T08:39:48.372 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:39:48.374 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-09T08:39:48.395 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-09T08:39:48.397 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:48.422 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:48 +0000 (0:00:00.101) 0:00:08.603 ********* 2021-03-09T08:39:48.492 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-09T08:39:48.494 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:39:48.515 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:48 +0000 (0:00:00.092) 0:00:08.695 ********* 2021-03-09T08:39:48.564 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:39:48.566 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T08:39:48.753 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-03-09T08:39:48.755 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:39:48.776 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:39:48 +0000 (0:00:00.260) 0:00:08.956 ********* 2021-03-09T08:39:48.784 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:39:48.786 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T08:39:48.965 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T08:39:49.062 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-09T08:39:49.159 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T08:39:49.254 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-09T08:39:49.351 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T08:39:49.459 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-09T08:39:49.575 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T08:39:49.652 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-09T08:39:49.780 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T08:39:49.854 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-09T08:39:49.980 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T08:39:50.046 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-09T08:39:50.172 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T08:39:50.247 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-09T08:39:50.375 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T08:39:50.439 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-09T08:39:50.567 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T08:39:50.634 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-09T08:39:50.771 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T08:39:50.837 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-09T08:39:50.976 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T08:39:51.029 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-09T08:39:51.167 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T08:39:51.220 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-09T08:39:51.367 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T08:39:51.422 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-09T08:39:51.562 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T08:39:51.618 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-09T08:39:51.771 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T08:39:51.837 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-09T08:39:51.967 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T08:39:52.032 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-09T08:39:52.171 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T08:39:52.235 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-09T08:39:52.363 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T08:39:52.427 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-09T08:39:52.566 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T08:39:52.630 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-09T08:39:52.767 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T08:39:52.813 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-09T08:39:52.982 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T08:39:53.014 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-09T08:39:53.173 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T08:39:53.175 INFO:teuthology.task.ansible.out: 2021-03-09T08:39:53.207 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-09T08:39:53.375 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T08:39:53.397 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-09T08:39:53.568 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T08:39:53.621 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-09T08:39:53.780 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T08:39:53.813 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-09T08:39:54.004 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T08:39:54.048 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-09T08:39:54.222 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T08:39:54.244 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-09T08:39:54.458 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-09T08:39:54.470 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T08:39:54.687 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-09T08:39:54.712 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T08:39:54.901 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-09T08:39:54.937 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T08:39:55.129 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-09T08:39:55.142 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T08:39:55.358 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-09T08:39:55.371 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T08:39:55.552 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-09T08:39:55.563 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T08:39:55.742 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-09T08:39:55.765 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T08:39:55.933 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-09T08:39:55.955 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T08:39:56.135 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-09T08:39:56.147 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:39:56.326 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-09T08:39:56.338 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T08:39:56.518 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-09T08:39:56.530 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T08:39:56.710 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:39:56.732 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T08:39:56.901 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-09T08:39:56.923 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T08:39:57.102 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-09T08:39:57.125 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T08:39:57.293 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-09T08:39:57.316 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T08:39:57.486 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-09T08:39:57.512 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T08:39:57.689 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-09T08:39:57.723 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:39:57.882 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-09T08:39:57.916 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T08:39:58.076 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-09T08:39:58.121 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T08:39:58.284 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:39:58.327 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T08:39:58.478 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-09T08:39:58.533 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T08:39:58.672 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-09T08:39:58.739 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T08:39:58.866 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-09T08:39:58.930 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T08:39:59.058 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-09T08:39:59.132 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T08:39:59.260 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-09T08:39:59.335 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T08:39:59.453 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-09T08:39:59.528 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T08:39:59.654 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-09T08:39:59.729 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T08:39:59.847 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-09T08:39:59.922 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T08:40:00.050 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-09T08:40:00.124 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T08:40:00.261 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-09T08:40:00.325 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T08:40:00.467 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-09T08:40:00.567 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T08:40:00.710 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-09T08:40:00.743 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T08:40:00.916 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-09T08:40:00.971 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T08:40:01.132 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-09T08:40:01.190 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:40:01.352 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-09T08:40:01.397 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T08:40:01.584 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-09T08:40:01.619 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T08:40:01.790 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-09T08:40:01.824 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T08:40:02.005 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-09T08:40:02.030 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T08:40:02.213 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-09T08:40:02.226 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T08:40:02.418 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-09T08:40:02.430 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T08:40:02.622 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-09T08:40:02.647 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T08:40:02.830 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-09T08:40:02.863 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T08:40:03.068 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-09T08:40:03.081 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T08:40:03.277 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-09T08:40:03.299 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T08:40:03.483 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-09T08:40:03.496 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T08:40:03.667 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-09T08:40:03.669 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:03.701 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T08:40:03.871 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-09T08:40:03.904 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T08:40:04.063 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-09T08:40:04.096 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T08:40:04.268 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-09T08:40:04.290 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T08:40:04.462 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-09T08:40:04.495 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T08:40:04.660 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-09T08:40:04.693 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T08:40:04.856 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-09T08:40:04.895 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T08:40:05.087 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-09T08:40:05.119 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T08:40:05.330 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-09T08:40:05.376 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T08:40:05.550 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-09T08:40:05.620 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T08:40:05.771 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-09T08:40:05.836 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T08:40:05.984 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-09T08:40:06.053 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T08:40:06.193 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-09T08:40:06.271 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T08:40:06.421 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-09T08:40:06.486 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T08:40:06.624 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-09T08:40:06.698 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T08:40:06.815 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-09T08:40:06.900 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T08:40:07.016 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-09T08:40:07.091 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T08:40:07.208 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-09T08:40:07.296 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T08:40:07.414 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-09T08:40:07.489 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T08:40:07.606 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-09T08:40:07.691 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T08:40:07.808 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-09T08:40:07.884 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T08:40:08.002 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-09T08:40:08.089 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T08:40:08.197 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-09T08:40:08.283 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T08:40:08.389 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-09T08:40:08.476 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T08:40:08.593 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-09T08:40:08.667 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T08:40:08.784 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-09T08:40:08.869 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T08:40:08.974 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-09T08:40:09.061 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:40:09.168 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-09T08:40:09.275 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T08:40:09.370 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-09T08:40:09.477 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T08:40:09.561 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-09T08:40:09.678 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T08:40:09.753 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-09T08:40:09.869 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T08:40:09.955 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-09T08:40:10.066 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T08:40:10.162 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-09T08:40:10.259 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T08:40:10.365 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-09T08:40:10.461 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T08:40:10.567 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-09T08:40:10.653 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T08:40:10.769 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-09T08:40:10.844 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T08:40:10.961 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-09T08:40:11.046 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T08:40:11.163 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-09T08:40:11.237 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T08:40:11.355 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-09T08:40:11.429 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T08:40:11.550 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-09T08:40:11.595 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:11.621 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T08:40:11.783 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-09T08:40:11.827 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T08:40:11.987 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-09T08:40:12.034 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T08:40:12.192 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-09T08:40:12.235 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T08:40:12.383 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-09T08:40:12.436 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T08:40:12.572 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-09T08:40:12.635 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T08:40:12.773 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-09T08:40:12.775 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:12.827 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T08:40:12.964 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-09T08:40:13.018 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T08:40:13.159 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-09T08:40:13.213 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T08:40:13.349 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-09T08:40:13.403 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T08:40:13.551 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-09T08:40:13.606 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T08:40:13.741 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-09T08:40:13.816 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T08:40:13.943 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-09T08:40:14.038 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T08:40:14.155 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-09T08:40:14.250 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T08:40:14.355 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-09T08:40:14.440 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T08:40:14.545 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-09T08:40:14.640 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T08:40:14.744 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-09T08:40:14.829 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T08:40:14.934 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-09T08:40:15.029 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T08:40:15.135 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-09T08:40:15.222 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T08:40:15.327 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-09T08:40:15.328 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:15.412 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T08:40:15.519 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-09T08:40:15.604 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T08:40:15.711 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-09T08:40:15.807 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T08:40:15.917 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-09T08:40:16.044 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T08:40:16.145 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-09T08:40:16.300 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T08:40:16.374 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-09T08:40:16.535 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T08:40:16.671 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-09T08:40:16.695 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:16 +0000 (0:00:27.919) 0:00:36.875 ********* 2021-03-09T08:40:16.697 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:16.784 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-09T08:40:16.818 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:16 +0000 (0:00:00.123) 0:00:36.998 ********* 2021-03-09T08:40:16.827 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:40:16.828 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-09T08:40:16.927 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-09T08:40:16.928 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T08:40:16.930 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T08:40:16.932 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T08:40:16.933 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T08:40:16.935 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T08:40:16.936 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T08:40:16.938 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T08:40:16.940 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T08:40:16.941 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T08:40:16.943 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T08:40:16.945 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T08:40:16.947 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T08:40:16.948 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T08:40:16.970 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:16 +0000 (0:00:00.151) 0:00:37.150 ********* 2021-03-09T08:40:17.053 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-09T08:40:17.055 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-09T08:40:17.500 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:17 +0000 (0:00:00.529) 0:00:37.680 ********* 2021-03-09T08:40:17.582 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-09T08:40:17.611 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:17 +0000 (0:00:00.110) 0:00:37.791 ********* 2021-03-09T08:40:17.703 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-09T08:40:17.705 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-09T08:40:17.706 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-09T08:40:17.708 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-09T08:40:17.709 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-09T08:40:17.711 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5 2021-03-09T08:40:17.711 INFO:teuthology.task.ansible.out:mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-09T08:40:17.713 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-09T08:40:17.715 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-09T08:40:17.716 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-09T08:40:17.718 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-09T08:40:17.719 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-09T08:40:17.721 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-09T08:40:17.723 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-noteboo 2021-03-09T08:40:17.723 INFO:teuthology.task.ansible.out:k LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-09T08:40:17.725 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-09T08:40:17.752 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:17 +0000 (0:00:00.140) 0:00:37.932 ********* 2021-03-09T08:40:18.055 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-09T08:40:18.056 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T08:40:18.085 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:18 +0000 (0:00:00.332) 0:00:38.264 ********* 2021-03-09T08:40:18.418 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:40:18.420 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T08:40:18.431 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-09T08:40:18.433 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-09T08:40:18.467 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:18 +0000 (0:00:00.382) 0:00:38.647 ********* 2021-03-09T08:40:18.623 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:40:18.625 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T08:40:18.820 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-09T08:40:18.831 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T08:40:19.030 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-09T08:40:19.041 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-09T08:40:19.043 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-09T08:40:19.251 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T08:40:19.252 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-09T08:40:19.254 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:19.454 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-09T08:40:19.455 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y 2021-03-09T08:40:19.456 INFO:teuthology.task.ansible.out:+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-09T08:40:19.658 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-09T08:40:19.670 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T08:40:19.860 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-09T08:40:19.871 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-09T08:40:19.873 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-09T08:40:20.072 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-09T08:40:20.073 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:20.075 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-09T08:40:20.085 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:20.274 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T08:40:20.276 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-09T08:40:20.289 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-09T08:40:20.492 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-09T08:40:20.493 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T08:40:20.495 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-09T08:40:20.505 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:20.507 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:20.517 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:20.696 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-09T08:40:20.698 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-09T08:40:20.719 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-09T08:40:20.898 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-09T08:40:20.900 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-09T08:40:20.910 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:20.921 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:20.922 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:20.934 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T08:40:20.935 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T08:40:21.125 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-09T08:40:21.136 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T08:40:21.327 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-09T08:40:21.329 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-09T08:40:21.360 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-09T08:40:21.362 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-09T08:40:21.539 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-09T08:40:21.562 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-09T08:40:21.741 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-09T08:40:21.743 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-09T08:40:21.764 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T08:40:21.942 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-09T08:40:21.975 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T08:40:22.152 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-09T08:40:22.174 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-09T08:40:22.176 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-09T08:40:22.353 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-09T08:40:22.385 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T08:40:22.387 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-09T08:40:22.552 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-09T08:40:22.554 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-09T08:40:22.595 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T08:40:22.764 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-09T08:40:22.765 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-09T08:40:22.796 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T08:40:22.989 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-09T08:40:23.025 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T08:40:23.026 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T08:40:23.206 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-09T08:40:23.303 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T08:40:23.454 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-09T08:40:23.456 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-09T08:40:23.520 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T08:40:23.682 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-09T08:40:23.737 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T08:40:23.906 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-09T08:40:23.939 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T08:40:24.137 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-09T08:40:24.160 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T08:40:24.354 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-09T08:40:24.389 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T08:40:24.391 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-09T08:40:24.576 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-09T08:40:24.588 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T08:40:24.792 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-09T08:40:24.794 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T08:40:25.019 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T08:40:25.021 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-09T08:40:25.265 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-09T08:40:25.268 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T08:40:25.502 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-09T08:40:25.524 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T08:40:25.706 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-09T08:40:25.729 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T08:40:25.917 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-09T08:40:25.929 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:40:26.118 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-09T08:40:26.140 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T08:40:26.329 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:40:26.351 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T08:40:26.541 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-09T08:40:26.553 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T08:40:26.554 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T08:40:26.746 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-09T08:40:26.758 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T08:40:26.759 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T08:40:26.939 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-09T08:40:26.941 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-09T08:40:26.961 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T08:40:26.963 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:27.162 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-09T08:40:27.163 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-09T08:40:27.173 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T08:40:27.385 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-09T08:40:27.418 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T08:40:27.667 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T08:40:27.669 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:40:27.678 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-09T08:40:27.926 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-09T08:40:27.927 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:27.929 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T08:40:27.930 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T08:40:27.932 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T08:40:28.181 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T08:40:28.194 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-09T08:40:28.196 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:40:28.397 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T08:40:28.419 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-09T08:40:28.421 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-09T08:40:28.422 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-09T08:40:28.643 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-09T08:40:28.669 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T08:40:28.868 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-09T08:40:28.880 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T08:40:29.082 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-09T08:40:29.084 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T08:40:29.283 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T08:40:29.284 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-09T08:40:29.495 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-09T08:40:29.497 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T08:40:29.696 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-09T08:40:29.698 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-09T08:40:29.718 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T08:40:29.919 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-09T08:40:29.921 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T08:40:29.922 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-09T08:40:30.121 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-09T08:40:30.132 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T08:40:30.334 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-09T08:40:30.336 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T08:40:30.537 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-09T08:40:30.539 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T08:40:30.540 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-09T08:40:30.751 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-09T08:40:30.752 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-09T08:40:30.953 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T08:40:30.964 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-09T08:40:30.966 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-09T08:40:31.155 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T08:40:31.166 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-09T08:40:31.359 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:40:31.381 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-09T08:40:31.565 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T08:40:31.587 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-09T08:40:31.588 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:31.778 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-09T08:40:31.790 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-09T08:40:31.984 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-09T08:40:32.006 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-09T08:40:32.198 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-09T08:40:32.213 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-09T08:40:32.410 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-09T08:40:32.432 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-09T08:40:32.612 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-09T08:40:32.635 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-09T08:40:32.813 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T08:40:32.835 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-09T08:40:33.013 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T08:40:33.047 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-09T08:40:33.220 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T08:40:33.242 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-09T08:40:33.424 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T08:40:33.446 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-09T08:40:33.626 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T08:40:33.648 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-09T08:40:33.830 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T08:40:33.863 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-09T08:40:34.030 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T08:40:34.062 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-09T08:40:34.233 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T08:40:34.266 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-09T08:40:34.436 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T08:40:34.469 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-09T08:40:34.648 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T08:40:34.672 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-09T08:40:34.886 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-09T08:40:34.897 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T08:40:34.899 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T08:40:35.118 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T08:40:35.120 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-09T08:40:35.321 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T08:40:35.323 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-09T08:40:35.325 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-09T08:40:35.521 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T08:40:35.523 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-09T08:40:35.524 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:35.726 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-09T08:40:35.738 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T08:40:35.933 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-09T08:40:35.944 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T08:40:35.946 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-09T08:40:36.145 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T08:40:36.147 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/ 2021-03-09T08:40:36.148 INFO:teuthology.task.ansible.out:1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-09T08:40:36.348 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-09T08:40:36.350 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T08:40:36.558 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-09T08:40:36.560 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-09T08:40:36.572 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T08:40:36.791 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-09T08:40:36.813 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T08:40:37.034 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-09T08:40:37.056 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-09T08:40:37.247 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-09T08:40:37.259 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-09T08:40:37.449 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-09T08:40:37.460 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T08:40:37.661 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-09T08:40:37.673 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T08:40:37.863 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-09T08:40:37.885 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T08:40:37.887 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-09T08:40:38.075 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-09T08:40:38.086 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T08:40:38.299 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-09T08:40:38.301 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksir 2021-03-09T08:40:38.303 INFO:teuthology.task.ansible.out:ivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T08:40:38.507 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-09T08:40:38.519 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-09T08:40:38.811 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T08:40:38.828 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-09T08:40:39.074 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOO 2021-03-09T08:40:39.079 INFO:teuthology.task.ansible.out:b+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T08:40:39.294 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-09T08:40:39.326 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T08:40:39.526 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-09T08:40:39.569 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T08:40:39.572 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:40:39.752 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-09T08:40:39.850 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T08:40:39.976 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-09T08:40:39.978 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-09T08:40:40.071 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T08:40:40.198 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-09T08:40:40.283 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T08:40:40.400 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-09T08:40:40.495 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T08:40:40.611 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-09T08:40:40.699 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-09T08:40:40.815 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-09T08:40:40.901 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T08:40:41.027 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-09T08:40:41.102 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T08:40:41.227 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-09T08:40:41.301 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T08:40:41.427 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-09T08:40:41.429 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:41.502 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T08:40:41.628 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-09T08:40:41.703 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T08:40:41.833 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-09T08:40:41.918 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T08:40:42.035 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-09T08:40:42.121 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T08:40:42.269 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-09T08:40:42.323 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T08:40:42.470 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-09T08:40:42.534 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T08:40:42.682 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-09T08:40:42.736 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T08:40:42.883 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-09T08:40:42.936 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T08:40:43.084 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-09T08:40:43.149 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T08:40:43.307 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-09T08:40:43.350 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T08:40:43.508 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-09T08:40:43.561 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T08:40:43.563 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T08:40:43.709 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-09T08:40:43.762 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T08:40:43.911 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-09T08:40:43.913 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-09T08:40:43.964 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T08:40:44.122 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-09T08:40:44.176 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T08:40:44.324 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-09T08:40:44.367 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T08:40:44.547 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-09T08:40:44.569 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T08:40:44.749 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-09T08:40:45.012 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7T 2021-03-09T08:40:45.014 INFO:teuthology.task.ansible.out:chsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-09T08:40:45.037 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T08:40:45.184 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-09T08:40:45.269 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T08:40:45.270 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:45.386 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-09T08:40:45.470 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T08:40:45.596 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-09T08:40:45.680 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T08:40:45.807 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-09T08:40:45.886 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T08:40:46.012 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-09T08:40:46.087 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T08:40:46.207 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-09T08:40:46.293 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T08:40:46.419 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-09T08:40:46.494 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T08:40:46.631 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-09T08:40:46.695 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T08:40:46.849 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-09T08:40:46.915 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T08:40:47.065 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-09T08:40:47.156 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options" 2021-03-09T08:40:47.158 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-09T08:40:47.304 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-09T08:40:47.400 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T08:40:47.500 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:47 +0000 (0:00:29.032) 0:01:07.680 ********* 2021-03-09T08:40:47.531 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options" 2021-03-09T08:40:47.532 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-09T08:40:47.606 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.607 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.617 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.628 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.631 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.640 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.651 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.652 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.662 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.736 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-09T08:40:47.738 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-09T08:40:47.851 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:47 +0000 (0:00:00.351) 0:01:08.031 ********* 2021-03-09T08:40:47.869 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-09T08:40:47.871 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-09T08:40:47.872 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.883 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.947 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.968 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.970 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.981 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.983 INFO:teuthology.task.ansible.out: skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:47.994 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.005 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.008 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.090 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-09T08:40:48.230 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-09T08:40:48.252 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.263 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.296 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-09T08:40:48.307 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.318 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.329 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.331 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.341 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.352 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.354 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.364 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.375 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.377 INFO:teuthology.task.ansible.out: skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.387 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.388 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.399 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.410 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.412 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.422 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.433 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.435 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.445 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.455 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.457 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.467 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-09T08:40:48.469 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.479 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.481 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.490 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.501 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.512 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.514 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.524 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.526 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.535 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.546 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.549 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.558 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.569 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.571 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.580 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.595 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.603 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.614 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.615 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.625 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.636 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.638 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.647 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.659 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.661 INFO:teuthology.task.ansible.out: skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.662 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-09T08:40:48.672 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.674 INFO:teuthology.task.ansible.out: skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.675 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454 2021-03-09T08:40:48.676 INFO:teuthology.task.ansible.out:742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.686 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.687 INFO:teuthology.task.ansible.out: skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.698 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.700 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.701 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.712 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.713 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.724 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.725 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e1 2021-03-09T08:40:48.727 INFO:teuthology.task.ansible.out:8c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.736 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.739 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.750 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.751 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.761 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.763 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.773 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.775 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.787 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.789 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.799 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.801 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.803 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.805 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.815 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.817 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.828 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.829 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.840 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.841 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.852 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.854 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.864 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.866 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.877 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.878 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.888 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.890 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.890 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.901 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.903 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.915 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.916 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.927 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.929 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.939 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.940 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.940 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.952 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.953 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.965 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.976 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.977 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.988 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:48.990 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.000 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.002 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.012 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.014 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.025 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.027 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.037 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.039 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.049 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.050 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.051 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.051 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.052 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.063 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.065 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.066 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.077 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.079 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.089 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.100 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.102 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.113 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.113 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.114 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.116 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.127 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.137 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.139 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.150 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.152 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.163 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.165 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.175 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.177 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.188 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.190 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.220 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.221 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.243 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:49 +0000 (0:00:01.391) 0:01:09.423 ********* 2021-03-09T08:40:49.250 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.251 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.253 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.255 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.267 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.269 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.279 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.281 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.291 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.302 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.304 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.314 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.315 INFO:teuthology.task.ansible.out: skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.326 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-09T08:40:49.328 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.355 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:49 +0000 (0:00:00.112) 0:01:09.535 ********* 2021-03-09T08:40:49.365 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.366 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.368 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.369 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.380 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.382 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.392 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.403 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.405 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.415 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.417 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.428 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.438 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.441 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.450 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.462 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.472 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.484 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.495 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.497 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.506 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.520 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.521 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.534 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.537 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.548 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.550 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.561 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.564 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.577 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.590 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.592 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.594 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.612 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.613 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@thinkpad 5H//Tz6a0uBQDJiYlp9bIA 5ad4213c5f177c9d3586e58770208dd3e18f57d78f12d5d1a9dfa541d5c5c957\n"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.626 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.689 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:49 +0000 (0:00:00.334) 0:01:09.870 ********* 2021-03-09T08:40:49.704 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:40:49.705 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-09T08:40:49.822 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-09T08:40:49.824 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:40:49.854 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:49 +0000 (0:00:00.164) 0:01:10.034 ********* 2021-03-09T08:40:49.932 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-09T08:40:49.933 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-09T08:40:50.132 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T08:40:50.249 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-09T08:40:50.375 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-09T08:40:50.377 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:50.463 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-09T08:40:50.569 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T08:40:50.699 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-09T08:40:50.775 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T08:40:50.929 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-09T08:40:50.972 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T08:40:51.121 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-09T08:40:51.164 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T08:40:51.165 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:51.322 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-09T08:40:51.324 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:51.355 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T08:40:51.513 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-09T08:40:51.546 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T08:40:51.705 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-09T08:40:51.738 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T08:40:51.907 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-09T08:40:51.929 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T08:40:52.086 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-09T08:40:52.119 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T08:40:52.288 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-09T08:40:52.310 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T08:40:52.479 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-09T08:40:52.501 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T08:40:52.671 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-09T08:40:52.705 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T08:40:52.863 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-09T08:40:52.895 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T08:40:53.060 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-09T08:40:53.094 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T08:40:53.260 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-09T08:40:53.315 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T08:40:53.454 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-09T08:40:53.518 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T08:40:53.680 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-09T08:40:53.725 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T08:40:53.880 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-09T08:40:53.946 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T08:40:54.097 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-09T08:40:54.161 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T08:40:54.312 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-09T08:40:54.345 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T08:40:54.526 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-09T08:40:54.538 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T08:40:54.726 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-09T08:40:54.738 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T08:40:54.917 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-09T08:40:54.929 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T08:40:54.930 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:55.109 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-09T08:40:55.121 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T08:40:55.313 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-09T08:40:55.314 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T08:40:55.507 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-09T08:40:55.508 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T08:40:55.701 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-09T08:40:55.713 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T08:40:55.904 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-09T08:40:55.918 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T08:40:56.101 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-09T08:40:56.124 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T08:40:56.306 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-09T08:40:56.308 INFO:teuthology.task.ansible.out: ok: [smithi099.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T08:40:56.498 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-09T08:40:56.509 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T08:40:56.710 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-09T08:40:56.732 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T08:40:56.902 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-09T08:40:56.925 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T08:40:57.104 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-09T08:40:57.116 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T08:40:57.303 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-09T08:40:57.315 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T08:40:57.494 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-09T08:40:57.496 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T08:40:57.686 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} ok: [smithi195.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-09T08:40:57.875 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T08:40:57.876 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-09T08:40:58.077 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T08:40:58.080 INFO:teuthology.task.ansible.out: ok: [smithi195.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-09T08:40:58.081 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:58.270 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T08:40:58.282 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-09T08:40:58.463 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T08:40:58.474 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-09T08:40:58.659 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T08:40:58.672 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-09T08:40:58.877 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T08:40:58.878 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:58.902 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-09T08:40:58.904 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:59.095 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T08:40:59.096 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:59.129 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-09T08:40:59.288 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T08:40:59.321 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-09T08:40:59.350 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:59 +0000 (0:00:09.496) 0:01:19.530 ********* 2021-03-09T08:40:59.351 INFO:teuthology.task.ansible.out: 2021-03-09T08:40:59.510 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-09T08:40:59.703 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-09T08:40:59.915 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-09T08:40:59.978 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:40:59 +0000 (0:00:00.627) 0:01:20.158 ********* 2021-03-09T08:40:59.987 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279259.4580746-10452-139344813789262/source", "state": "file", "uid": 0} 2021-03-09T08:41:00.016 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:00 +0000 (0:00:00.038) 0:01:20.196 ********* 2021-03-09T08:41:00.140 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-09T08:41:00.142 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => { "msg": "Host smithi099.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-03-09T08:41:00.166 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:00 +0000 (0:00:00.150) 0:01:20.347 ********* 2021-03-09T08:41:00.258 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:00.260 INFO:teuthology.task.ansible.out:TASK [common : Including major version specific variables.] ******************** ok: [smithi099.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-03-09T08:41:00.284 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:00 +0000 (0:00:00.117) 0:01:20.464 ********* 2021-03-09T08:41:00.286 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:00.368 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:00.370 INFO:teuthology.task.ansible.out:TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:00.401 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:00 +0000 (0:00:00.117) 0:01:20.582 ********* 2021-03-09T08:41:00.591 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-09T08:41:00.593 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279260.0913084-10507-238411611306582/source", "state": "file", "uid": 0} 2021-03-09T08:41:00.616 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:00 +0000 (0:00:00.214) 0:01:20.796 ********* 2021-03-09T08:41:00.697 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-09T08:41:00.699 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => { "msg": "Host smithi195.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-03-09T08:41:00.729 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:00 +0000 (0:00:00.113) 0:01:20.909 ********* 2021-03-09T08:41:00.834 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-09T08:41:00.838 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-03-09T08:41:00.869 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:00 +0000 (0:00:00.139) 0:01:21.049 ********* 2021-03-09T08:41:01.021 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:01.024 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.352519", "end": "2021-03-09 08:41:00.983052", "rc": 0, "start": "2021-03-09 08:41:00.630533", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-09T08:41:01.045 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:01 +0000 (0:00:00.176) 0:01:21.225 ********* 2021-03-09T08:41:01.096 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:01 +0000 (0:00:00.050) 0:01:21.275 ********* 2021-03-09T08:41:01.195 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-09T08:41:01.197 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:01.235 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:01 +0000 (0:00:00.138) 0:01:21.414 ********* 2021-03-09T08:41:01.398 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-09T08:41:01.399 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:01.434 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:01 +0000 (0:00:00.199) 0:01:21.614 ********* 2021-03-09T08:41:01.550 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-09T08:41:01.552 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:01.578 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:01 +0000 (0:00:00.141) 0:01:21.756 ********* 2021-03-09T08:41:01.718 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-09T08:41:01.721 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:01.746 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:01 +0000 (0:00:00.169) 0:01:21.925 ********* 2021-03-09T08:41:01.806 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:01.808 INFO:teuthology.task.ansible.out:TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.354939", "end": "2021-03-09 08:41:01.781066", "rc": 0, "start": "2021-03-09 08:41:01.426127", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-09T08:41:01.831 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:01 +0000 (0:00:00.086) 0:01:22.011 ********* 2021-03-09T08:41:01.871 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:01.923 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:01 +0000 (0:00:00.091) 0:01:22.103 ********* 2021-03-09T08:41:01.970 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:02.007 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:01 +0000 (0:00:00.082) 0:01:22.186 ********* 2021-03-09T08:41:02.088 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:02.116 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.110) 0:01:22.296 ********* 2021-03-09T08:41:02.135 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-09T08:41:02.138 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:02.184 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.067) 0:01:22.364 ********* 2021-03-09T08:41:02.245 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-09T08:41:02.246 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:02.286 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.101) 0:01:22.465 ********* 2021-03-09T08:41:02.343 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi099.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:02.346 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:02.391 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.105) 0:01:22.571 ********* 2021-03-09T08:41:02.414 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-09T08:41:02.416 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:02.450 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.059) 0:01:22.631 ********* 2021-03-09T08:41:02.503 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-09T08:41:02.505 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:02.542 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.091) 0:01:22.722 ********* 2021-03-09T08:41:02.588 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-09T08:41:02.590 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:02.627 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.084) 0:01:22.807 ********* 2021-03-09T08:41:02.685 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-09T08:41:02.687 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:02.713 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.086) 0:01:22.893 ********* 2021-03-09T08:41:02.764 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-09T08:41:02.773 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:02.791 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.077) 0:01:22.971 ********* 2021-03-09T08:41:02.808 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-09T08:41:02.810 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:02.842 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.051) 0:01:23.023 ********* 2021-03-09T08:41:02.884 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-09T08:41:02.885 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi195.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:02.909 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.066) 0:01:23.089 ********* 2021-03-09T08:41:02.982 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-09T08:41:02.984 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.006 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:02 +0000 (0:00:00.096) 0:01:23.186 ********* 2021-03-09T08:41:03.031 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-09T08:41:03.033 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.058 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.051) 0:01:23.238 ********* 2021-03-09T08:41:03.112 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-09T08:41:03.114 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:03.146 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.088) 0:01:23.326 ********* 2021-03-09T08:41:03.188 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-09T08:41:03.190 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.218 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.071) 0:01:23.398 ********* 2021-03-09T08:41:03.258 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-09T08:41:03.260 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.283 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.065) 0:01:23.463 ********* 2021-03-09T08:41:03.303 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-09T08:41:03.305 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:03.329 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.046) 0:01:23.510 ********* 2021-03-09T08:41:03.358 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-09T08:41:03.360 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.382 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.052) 0:01:23.562 ********* 2021-03-09T08:41:03.411 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-09T08:41:03.413 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.440 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.057) 0:01:23.620 ********* 2021-03-09T08:41:03.458 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-09T08:41:03.460 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.484 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.044) 0:01:23.664 ********* 2021-03-09T08:41:03.523 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-09T08:41:03.525 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:03.549 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.064) 0:01:23.729 ********* 2021-03-09T08:41:03.568 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T08:41:03.569 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.594 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.044) 0:01:23.774 ********* 2021-03-09T08:41:03.635 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-09T08:41:03.637 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.672 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.078) 0:01:23.852 ********* 2021-03-09T08:41:03.681 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-09T08:41:03.682 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:03.709 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.037) 0:01:23.890 ********* 2021-03-09T08:41:03.755 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-09T08:41:03.757 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.783 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.073) 0:01:23.963 ********* 2021-03-09T08:41:03.812 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T08:41:03.814 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.837 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.053) 0:01:24.017 ********* 2021-03-09T08:41:03.886 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-09T08:41:03.888 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.912 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.074) 0:01:24.091 ********* 2021-03-09T08:41:03.918 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-09T08:41:03.920 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:03.943 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:03 +0000 (0:00:00.031) 0:01:24.123 ********* 2021-03-09T08:41:03.988 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-09T08:41:03.990 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.015 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.072) 0:01:24.195 ********* 2021-03-09T08:41:04.033 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-09T08:41:04.035 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.062 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.046) 0:01:24.242 ********* 2021-03-09T08:41:04.091 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-09T08:41:04.093 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:04.117 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.055) 0:01:24.297 ********* 2021-03-09T08:41:04.146 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-09T08:41:04.148 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.170 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.053) 0:01:24.350 ********* 2021-03-09T08:41:04.188 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T08:41:04.189 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.214 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.044) 0:01:24.395 ********* 2021-03-09T08:41:04.242 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-09T08:41:04.244 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.266 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.051) 0:01:24.446 ********* 2021-03-09T08:41:04.297 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-09T08:41:04.299 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.322 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.056) 0:01:24.503 ********* 2021-03-09T08:41:04.341 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-09T08:41:04.342 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.364 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.041) 0:01:24.545 ********* 2021-03-09T08:41:04.413 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-09T08:41:04.415 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.438 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.073) 0:01:24.618 ********* 2021-03-09T08:41:04.455 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-09T08:41:04.457 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:04.478 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.040) 0:01:24.659 ********* 2021-03-09T08:41:04.517 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-09T08:41:04.519 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.547 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.068) 0:01:24.727 ********* 2021-03-09T08:41:04.566 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-09T08:41:04.568 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.590 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.043) 0:01:24.771 ********* 2021-03-09T08:41:04.619 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-09T08:41:04.620 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.643 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.052) 0:01:24.823 ********* 2021-03-09T08:41:04.673 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-09T08:41:04.675 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.699 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.055) 0:01:24.879 ********* 2021-03-09T08:41:04.716 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-09T08:41:04.718 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.741 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.042) 0:01:24.921 ********* 2021-03-09T08:41:04.811 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-09T08:41:04.813 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-09T08:41:04.833 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.092) 0:01:25.013 ********* 2021-03-09T08:41:04.902 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-09T08:41:04.904 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:04.927 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.093) 0:01:25.107 ********* 2021-03-09T08:41:04.966 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-09T08:41:04.967 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:41:04.994 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:04 +0000 (0:00:00.067) 0:01:25.174 ********* 2021-03-09T08:41:05.022 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-09T08:41:05.024 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:05.050 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:05 +0000 (0:00:00.056) 0:01:25.230 ********* 2021-03-09T08:41:05.320 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-09T08:41:05.321 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:41:05.343 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:05 +0000 (0:00:00.292) 0:01:25.523 ********* 2021-03-09T08:41:05.803 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-09T08:41:05.805 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-03-09T08:41:05.879 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-03-09T08:41:06.249 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-03-09T08:41:06.279 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:06 +0000 (0:00:00.936) 0:01:26.459 ********* 2021-03-09T08:41:06.337 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-03-09T08:41:06.368 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:06 +0000 (0:00:00.089) 0:01:26.549 ********* 2021-03-09T08:41:06.825 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-09T08:41:06.827 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-03-09T08:41:06.922 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-03-09T08:41:07.314 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-03-09T08:41:07.316 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:07.340 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.971) 0:01:27.521 ********* 2021-03-09T08:41:07.372 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-03-09T08:41:07.374 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:07.398 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.057) 0:01:27.578 ********* 2021-03-09T08:41:07.426 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-09T08:41:07.428 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.451 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.053) 0:01:27.632 ********* 2021-03-09T08:41:07.480 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.511 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.059) 0:01:27.691 ********* 2021-03-09T08:41:07.531 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T08:41:07.532 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.554 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.043) 0:01:27.734 ********* 2021-03-09T08:41:07.593 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.617 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.062) 0:01:27.797 ********* 2021-03-09T08:41:07.635 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-09T08:41:07.636 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.659 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.041) 0:01:27.839 ********* 2021-03-09T08:41:07.698 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.723 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.063) 0:01:27.903 ********* 2021-03-09T08:41:07.744 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-09T08:41:07.746 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.769 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.044) 0:01:27.949 ********* 2021-03-09T08:41:07.798 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.822 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.052) 0:01:28.002 ********* 2021-03-09T08:41:07.872 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-09T08:41:07.874 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.875 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.910 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.088) 0:01:28.090 ********* 2021-03-09T08:41:07.938 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.940 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:07.964 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:07 +0000 (0:00:00.053) 0:01:28.144 ********* 2021-03-09T08:41:08.014 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-09T08:41:08.553 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.084) 0:01:28.229 ********* skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.094) 0:01:28.323 ********* TASK [common : Set the timezone] *********************************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.117) 0:01:28.441 ********* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.089) 0:01:28.530 ********* TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.061) 0:01:28.592 ********* TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.137) 0:01:28.729 ********* 2021-03-09T08:41:08.577 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-09T08:41:08.578 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:08.604 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.055) 0:01:28.785 ********* 2021-03-09T08:41:08.670 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:08.693 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.088) 0:01:28.873 ********* 2021-03-09T08:41:08.724 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-09T08:41:08.727 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:08.771 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.077) 0:01:28.951 ********* 2021-03-09T08:41:08.873 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-09T08:41:08.876 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:08.896 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:08 +0000 (0:00:00.124) 0:01:29.076 ********* 2021-03-09T08:41:09.527 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-09T08:41:09.529 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.599464", "end": "2021-03-09 08:41:09.500356", "rc": 0, "start": "2021-03-09 08:41:08.900892", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.433551", "end": "2021-03-09 08:41:09.500335", "rc": 0, "start": "2021-03-09 08:41:09.066784", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:41:09.567 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:09 +0000 (0:00:00.670) 0:01:29.747 ********* 2021-03-09T08:41:09.600 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:09 +0000 (0:00:00.032) 0:01:29.780 ********* 2021-03-09T08:41:13.965 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:13.991 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:13 +0000 (0:00:04.388) 0:01:34.168 ********* 2021-03-09T08:41:14.084 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:14.117 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:14 +0000 (0:00:00.128) 0:01:34.297 ********* 2021-03-09T08:41:14.136 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-09T08:41:14.139 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:14.182 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:14 +0000 (0:00:00.064) 0:01:34.362 ********* 2021-03-09T08:41:14.214 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-09T08:41:14.218 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:14.242 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:14 +0000 (0:00:00.059) 0:01:34.421 ********* 2021-03-09T08:41:14.323 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-09T08:41:14.325 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:14.356 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:14 +0000 (0:00:00.115) 0:01:34.537 ********* 2021-03-09T08:41:14.365 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-09T08:41:14.367 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:14.417 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:14 +0000 (0:00:00.060) 0:01:34.597 ********* 2021-03-09T08:41:14.492 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-09T08:41:14.495 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:14.527 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:14 +0000 (0:00:00.110) 0:01:34.707 ********* 2021-03-09T08:41:14.566 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:14.568 INFO:teuthology.task.ansible.out:TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:14.595 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:14 +0000 (0:00:00.067) 0:01:34.775 ********* 2021-03-09T08:41:14.635 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-09T08:41:14.637 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:14.680 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:14 +0000 (0:00:00.084) 0:01:34.859 ********* 2021-03-09T08:41:14.758 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-09T08:41:14.760 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:14.795 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:14 +0000 (0:00:00.116) 0:01:34.976 ********* 2021-03-09T08:41:15.207 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-09T08:41:15.209 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-03-09T08:41:15.233 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:15 +0000 (0:00:00.437) 0:01:35.413 ********* 2021-03-09T08:41:15.378 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-03-09T08:41:15.403 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:15 +0000 (0:00:00.169) 0:01:35.583 ********* 2021-03-09T08:41:16.624 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-09T08:41:16.626 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:16.648 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:16 +0000 (0:00:01.244) 0:01:36.828 ********* 2021-03-09T08:41:16.761 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:16.789 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:16 +0000 (0:00:00.141) 0:01:36.970 ********* 2021-03-09T08:41:16.791 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:17.199 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-09T08:41:17.201 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T08:41:17.328 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-03-09T08:41:17.642 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T08:41:17.779 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-03-09T08:41:18.089 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-03-09T08:41:18.117 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:18 +0000 (0:00:01.327) 0:01:38.297 ********* 2021-03-09T08:41:18.231 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-03-09T08:41:18.270 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:18 +0000 (0:00:00.152) 0:01:38.450 ********* 2021-03-09T08:41:18.320 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-09T08:41:18.322 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:41:18.345 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:18 +0000 (0:00:00.075) 0:01:38.525 ********* 2021-03-09T08:41:18.458 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:41:18.481 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:18 +0000 (0:00:00.135) 0:01:38.661 ********* 2021-03-09T08:41:18.858 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:41:18.860 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-03-09T08:41:18.985 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-03-09T08:41:19.279 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-03-09T08:41:19.395 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-03-09T08:41:19.689 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-03-09T08:41:19.716 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:19 +0000 (0:00:01.235) 0:01:39.896 ********* 2021-03-09T08:41:19.796 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T08:41:19.798 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:19.823 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:19 +0000 (0:00:00.106) 0:01:40.003 ********* 2021-03-09T08:41:19.831 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-09T08:41:19.831 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-03-09T08:41:19.860 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:19 +0000 (0:00:00.037) 0:01:40.040 ********* 2021-03-09T08:41:19.899 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-09T08:41:19.901 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-03-09T08:41:19.927 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:19 +0000 (0:00:00.066) 0:01:40.107 ********* 2021-03-09T08:41:19.945 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-09T08:41:19.947 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:19.969 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:19 +0000 (0:00:00.042) 0:01:40.149 ********* 2021-03-09T08:41:20.060 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-09T08:41:20.063 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-03-09T08:41:20.088 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:20 +0000 (0:00:00.118) 0:01:40.268 ********* 2021-03-09T08:41:20.163 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T08:41:20.165 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-03-09T08:41:20.189 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:20 +0000 (0:00:00.101) 0:01:40.369 ********* 2021-03-09T08:41:20.271 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T08:41:20.272 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:20.282 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:20.304 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:20.305 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:20.327 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:20 +0000 (0:00:00.138) 0:01:40.507 ********* 2021-03-09T08:41:21.337 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-09T08:41:21.340 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-03-09T08:41:21.361 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:21 +0000 (0:00:01.034) 0:01:41.541 ********* 2021-03-09T08:41:21.442 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-09T08:41:21.444 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:21.464 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:21.466 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:21.489 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:21 +0000 (0:00:00.127) 0:01:41.669 ********* 2021-03-09T08:41:21.694 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-09T08:41:21.696 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:21.717 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:21 +0000 (0:00:00.227) 0:01:41.897 ********* 2021-03-09T08:41:21.797 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-09T08:41:21.799 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:21.819 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:21 +0000 (0:00:00.101) 0:01:41.999 ********* 2021-03-09T08:41:22.332 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-09T08:41:22.334 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-03-09T08:41:22.359 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:22 +0000 (0:00:00.539) 0:01:42.539 ********* 2021-03-09T08:41:22.869 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-09T08:41:22.871 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:22.894 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:22 +0000 (0:00:00.534) 0:01:43.074 ********* 2021-03-09T08:41:22.901 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-09T08:41:22.902 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-03-09T08:41:22.925 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:22 +0000 (0:00:00.031) 0:01:43.106 ********* 2021-03-09T08:41:22.964 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-09T08:41:22.966 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:22.992 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:22 +0000 (0:00:00.066) 0:01:43.172 ********* 2021-03-09T08:41:23.025 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:23.027 INFO:teuthology.task.ansible.out:TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi099.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:23.051 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:23 +0000 (0:00:00.059) 0:01:43.231 ********* 2021-03-09T08:41:23.384 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-09T08:41:23.386 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:41:23.412 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:23 +0000 (0:00:00.361) 0:01:43.592 ********* 2021-03-09T08:41:23.612 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-09T08:41:23.613 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-03-09T08:41:23.642 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:23 +0000 (0:00:00.230) 0:01:43.823 ********* 2021-03-09T08:41:23.744 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-09T08:41:23.745 INFO:teuthology.task.ansible.out:fatal: [smithi099.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-03-09T08:41:23.769 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:23 +0000 (0:00:00.125) 0:01:43.949 ********* 2021-03-09T08:41:23.848 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-09T08:41:23.850 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:23.871 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:23 +0000 (0:00:00.102) 0:01:44.051 ********* 2021-03-09T08:41:24.170 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-03-09T08:41:24.194 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:24 +0000 (0:00:00.323) 0:01:44.374 ********* 2021-03-09T08:41:24.275 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-09T08:41:24.277 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:24.287 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-03-09T08:41:24.309 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:24 +0000 (0:00:00.114) 0:01:44.489 ********* 2021-03-09T08:41:24.390 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-09T08:41:24.392 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-03-09T08:41:24.423 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:24 +0000 (0:00:00.114) 0:01:44.603 ********* 2021-03-09T08:41:24.559 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-09T08:41:24.561 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:41:24.588 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:24 +0000 (0:00:00.164) 0:01:44.768 ********* 2021-03-09T08:41:24.889 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:24.891 INFO:teuthology.task.ansible.out:TASK [common : Check firewalld status] ***************************************** fatal: [smithi195.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-03-09T08:41:24.916 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:24 +0000 (0:00:00.328) 0:01:45.096 ********* 2021-03-09T08:41:24.990 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:41:25.013 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:25 +0000 (0:00:00.097) 0:01:45.193 ********* 2021-03-09T08:41:25.306 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-09T08:41:25.310 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:39:06 UTC", "ActiveEnterTimestampMonotonic": "12891829", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service basic.target network.target local-fs.target time-sync.target system.slice nss-lookup.target tmp.mount var-run.mount sysinit.target -.mount systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:39:06 UTC", "AssertTimestampMonotonic": "12890952", "Before": "xdm.service getty@tty1.service shutdown.target plymouth-quit.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:39:06 UTC", "ConditionTimestampMonotonic": "12890951", "ConfigurationDirectoryMode": " 2021-03-09T08:41:25.312 INFO:teuthology.task.ansible.out:0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "960", "ExecMainStartTimestamp": "Tue 2021-03-09 08:39:06 UTC", "ExecMainStartTimestampMonotonic": "12891788", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Tue 2021-03-09 08:39:06 UTC] ; stop_time=[n/a] ; pid=960 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:39:06 UTC", "InactiveExitTimestampMonotonic": "12891829", "InvocationID": "0638a32771424c37af58072d38431223", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infin 2021-03-09T08:41:25.313 INFO:teuthology.task.ansible.out:ity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127252", "LimitNPROCSoft": "127252", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127252", "LimitSIGPENDINGSoft": "127252", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "960", "MemoryAccounting": "yes", "MemoryCurrent": "1626112", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualSta 2021-03-09T08:41:25.315 INFO:teuthology.task.ansible.out:rt": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:39:06 UTC", "StateChangeTimestampMonotonic": "12891829", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:39:06 UTC", "WatchdogTimestampMonotonic": "12891823", "WatchdogUSec": "0"}} 2021-03-09T08:41:25.347 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:25 +0000 (0:00:00.333) 0:01:45.527 ********* 2021-03-09T08:41:25.579 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-09T08:41:25.584 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-03-09T08:41:25.609 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:25 +0000 (0:00:00.260) 0:01:45.788 ********* 2021-03-09T08:41:25.647 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001669", "end": "2021-03-09 08:41:25.619596", "rc": 0, "start": "2021-03-09 08:41:25.617927", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-03-09T08:41:25.674 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:25 +0000 (0:00:00.066) 0:01:45.854 ********* 2021-03-09T08:41:26.015 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-09T08:41:26.018 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:39:07 UTC", "ActiveEnterTimestampMonotonic": "13347529", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount network.target systemd-journald.socket system.slice systemd-tmpfiles-setup.service nss-lookup.target local-fs.target time-sync.target sysinit.target basic.target var-run.mount -.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:39:07 UTC", "AssertTimestampMonotonic": "13346332", "Before": "multi-user.target xdm.service shutdown.target plymouth-quit.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:39:07 UTC", "ConditionTimestampMonotonic": "13346332", "ConfigurationDirectoryMode": " 2021-03-09T08:41:26.019 INFO:teuthology.task.ansible.out:0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "976", "ExecMainStartTimestamp": "Tue 2021-03-09 08:39:07 UTC", "ExecMainStartTimestampMonotonic": "13347465", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Tue 2021-03-09 08:39:07 UTC] ; stop_time=[n/a] ; pid=976 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:39:07 UTC", "InactiveExitTimestampMonotonic": "13347529", "InvocationID": "5d8a3374548d4a4291bc8f9cce2bf1d5", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infin 2021-03-09T08:41:26.021 INFO:teuthology.task.ansible.out:ity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127253", "LimitNPROCSoft": "127253", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127253", "LimitSIGPENDINGSoft": "127253", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "976", "MemoryAccounting": "yes", "MemoryCurrent": "1929216", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualSta 2021-03-09T08:41:26.022 INFO:teuthology.task.ansible.out:rt": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:39:07 UTC", "StateChangeTimestampMonotonic": "13347529", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:39:07 UTC", "WatchdogTimestampMonotonic": "13347528", "WatchdogUSec": "0"}} 2021-03-09T08:41:26.047 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:26 +0000 (0:00:00.372) 0:01:46.227 ********* 2021-03-09T08:41:26.314 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-09T08:41:26.315 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001735", "end": "2021-03-09 08:41:26.284225", "rc": 0, "start": "2021-03-09 08:41:26.282490", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-03-09T08:41:26.346 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:26 +0000 (0:00:00.299) 0:01:46.526 ********* 2021-03-09T08:41:27.118 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-09T08:41:27.120 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:27.143 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:27 +0000 (0:00:00.797) 0:01:47.323 ********* 2021-03-09T08:41:27.723 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:27.746 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:27 +0000 (0:00:00.602) 0:01:47.926 ********* 2021-03-09T08:41:28.507 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-09T08:41:28.509 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:28.531 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:28 +0000 (0:00:00.784) 0:01:48.711 ********* 2021-03-09T08:41:29.121 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:41:29.145 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:29 +0000 (0:00:00.613) 0:01:49.325 ********* 2021-03-09T08:41:29.508 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-03-09T08:41:29.531 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:29 +0000 (0:00:00.386) 0:01:49.712 ********* 2021-03-09T08:41:29.863 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-03-09T08:41:29.887 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:29 +0000 (0:00:00.355) 0:01:50.067 ********* 2021-03-09T08:41:46.040 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.278034", "end": "2021-03-09 08:41:46.018051", "failed_when_result": false, "rc": 0, "start": "2021-03-09 08:41:29.740017", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-03-09T08:41:46.063 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:46 +0000 (0:00:16.176) 0:02:06.243 ********* 2021-03-09T08:41:46.606 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-09T08:41:46.608 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-03-09T08:41:46.629 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:46 +0000 (0:00:00.565) 0:02:06.809 ********* 2021-03-09T08:41:46.648 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-09T08:41:46.649 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.531380", "end": "2021-03-09 08:41:46.624637", "failed_when_result": false, "rc": 0, "start": "2021-03-09 08:41:30.093257", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-03-09T08:41:46.675 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:46 +0000 (0:00:00.045) 0:02:06.855 ********* 2021-03-09T08:41:46.676 INFO:teuthology.task.ansible.out: 2021-03-09T08:41:46.896 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.018633", "end": "2021-03-09 08:41:46.873539", "rc": 0, "start": "2021-03-09 08:41:46.854906", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:41:46.922 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:46 +0000 (0:00:00.246) 0:02:07.102 ********* 2021-03-09T08:41:47.166 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-09T08:41:47.168 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.011015", "end": "2021-03-09 08:41:47.141751", "rc": 0, "start": "2021-03-09 08:41:47.130736", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:41:47.191 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:47 +0000 (0:00:00.269) 0:02:07.371 ********* 2021-03-09T08:41:47.211 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-09T08:41:47.213 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-03-09T08:41:47.239 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:47 +0000 (0:00:00.047) 0:02:07.419 ********* 2021-03-09T08:41:47.482 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-09T08:41:47.484 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.018992", "end": "2021-03-09 08:41:47.462822", "rc": 0, "start": "2021-03-09 08:41:47.443830", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:41:47.509 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:47 +0000 (0:00:00.270) 0:02:07.690 ********* 2021-03-09T08:41:47.803 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002432", "end": "2021-03-09 08:41:47.761852", "rc": 0, "start": "2021-03-09 08:41:47.759420", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:41:47.839 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:41:47 +0000 (0:00:00.328) 0:02:08.018 ********* 2021-03-09T08:42:03.690 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-09T08:42:03.692 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.262325", "end": "2021-03-09 08:42:03.658630", "rc": 0, "start": "2021-03-09 08:41:47.396305", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:03.716 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:03 +0000 (0:00:15.877) 0:02:23.896 ********* 2021-03-09T08:42:03.999 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-09T08:42:04.000 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-03-09T08:42:04.023 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.307) 0:02:24.204 ********* 2021-03-09T08:42:04.105 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-09T08:42:04.107 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-03-09T08:42:04.131 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.107) 0:02:24.311 ********* 2021-03-09T08:42:04.226 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-09T08:42:04.228 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T08:42:04.250 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.119) 0:02:24.430 ********* 2021-03-09T08:42:04.344 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-09T08:42:04.346 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-03-09T08:42:04.367 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.117) 0:02:24.548 ********* 2021-03-09T08:42:04.385 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-09T08:42:04.387 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.242554", "end": "2021-03-09 08:42:04.361776", "rc": 0, "start": "2021-03-09 08:41:48.119222", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:04.410 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.042) 0:02:24.590 ********* 2021-03-09T08:42:04.461 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-09T08:42:04.463 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T08:42:04.485 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.074) 0:02:24.665 ********* 2021-03-09T08:42:04.660 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-09T08:42:04.662 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-03-09T08:42:04.683 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.198) 0:02:24.864 ********* 2021-03-09T08:42:04.690 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-09T08:42:04.692 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:42:04.714 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.030) 0:02:24.894 ********* 2021-03-09T08:42:04.765 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-09T08:42:04.767 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-03-09T08:42:04.795 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.080) 0:02:24.975 ********* 2021-03-09T08:42:04.887 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-09T08:42:04.889 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T08:42:04.912 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.117) 0:02:25.092 ********* 2021-03-09T08:42:04.952 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T08:42:04.979 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:04 +0000 (0:00:00.067) 0:02:25.159 ********* 2021-03-09T08:42:04.998 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-09T08:42:04.999 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-03-09T08:42:05.036 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:05 +0000 (0:00:00.057) 0:02:25.217 ********* 2021-03-09T08:42:05.038 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:05.058 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:05.060 INFO:teuthology.task.ansible.out:TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:05.088 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:05 +0000 (0:00:00.051) 0:02:25.268 ********* 2021-03-09T08:42:05.140 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-09T08:42:05.142 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-09T08:42:05.165 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:05 +0000 (0:00:00.076) 0:02:25.345 ********* 2021-03-09T08:42:05.182 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-09T08:42:05.184 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:05.211 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:05 +0000 (0:00:00.045) 0:02:25.391 ********* 2021-03-09T08:42:05.212 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:05.357 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-09T08:42:05.359 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-03-09T08:42:05.383 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:05 +0000 (0:00:00.172) 0:02:25.564 ********* 2021-03-09T08:42:05.476 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-09T08:42:05.477 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-03-09T08:42:05.506 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:05 +0000 (0:00:00.122) 0:02:25.686 ********* 2021-03-09T08:42:05.629 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-09T08:42:05.631 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-09T08:42:05.670 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:05 +0000 (0:00:00.158) 0:02:25.844 ********* 2021-03-09T08:42:05.800 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:05.801 INFO:teuthology.task.ansible.out:TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:05.833 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:05 +0000 (0:00:00.168) 0:02:26.013 ********* 2021-03-09T08:42:05.928 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-09T08:42:05.930 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:05.955 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:05 +0000 (0:00:00.122) 0:02:26.136 ********* 2021-03-09T08:42:06.179 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-09T08:42:06.180 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-03-09T08:42:06.202 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:06 +0000 (0:00:00.246) 0:02:26.382 ********* 2021-03-09T08:42:06.220 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-09T08:42:06.222 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-03-09T08:42:06.246 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:06 +0000 (0:00:00.043) 0:02:26.426 ********* 2021-03-09T08:42:06.290 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-09T08:42:06.292 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi099"}, "changed": false} 2021-03-09T08:42:06.314 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:06 +0000 (0:00:00.068) 0:02:26.494 ********* 2021-03-09T08:42:06.862 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-09T08:42:06.864 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-03-09T08:42:06.886 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:06 +0000 (0:00:00.571) 0:02:27.066 ********* 2021-03-09T08:42:06.979 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-09T08:42:06.981 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi195"}, "changed": false} 2021-03-09T08:42:07.007 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:07 +0000 (0:00:00.121) 0:02:27.187 ********* 2021-03-09T08:42:07.389 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-09T08:42:07.391 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi099.front.sepia.ceph.com", "ansible_hostname": "smithi099", "ansible_nodename": "smithi099"}, "changed": true, "name": "smithi099"} 2021-03-09T08:42:07.415 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:07 +0000 (0:00:00.408) 0:02:27.595 ********* 2021-03-09T08:42:07.637 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi195.front.sepia.ceph.com", "ansible_hostname": "smithi195", "ansible_nodename": "smithi195"}, "changed": true, "name": "smithi195"} 2021-03-09T08:42:07.661 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:07 +0000 (0:00:00.246) 0:02:27.841 ********* 2021-03-09T08:42:07.922 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-09T08:42:07.923 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-03-09T08:42:07.946 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:07 +0000 (0:00:00.284) 0:02:28.126 ********* 2021-03-09T08:42:08.154 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-03-09T08:42:08.177 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:08 +0000 (0:00:00.231) 0:02:28.357 ********* 2021-03-09T08:42:08.438 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-09T08:42:08.439 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-03-09T08:42:08.462 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:08 +0000 (0:00:00.284) 0:02:28.642 ********* 2021-03-09T08:42:08.695 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-03-09T08:42:08.718 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:08 +0000 (0:00:00.255) 0:02:28.898 ********* 2021-03-09T08:42:09.043 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-09T08:42:09.044 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T08:42:09.077 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:09 +0000 (0:00:00.359) 0:02:29.258 ********* 2021-03-09T08:42:09.278 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-09T08:42:09.280 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:09.302 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:09 +0000 (0:00:00.224) 0:02:29.482 ********* 2021-03-09T08:42:09.592 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-09T08:42:09.594 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-03-09T08:42:09.615 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:09 +0000 (0:00:00.312) 0:02:29.795 ********* 2021-03-09T08:42:09.810 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-03-09T08:42:09.834 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:09 +0000 (0:00:00.219) 0:02:30.014 ********* 2021-03-09T08:42:09.854 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T08:42:09.856 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-03-09T08:42:09.878 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:09 +0000 (0:00:00.044) 0:02:30.059 ********* 2021-03-09T08:42:10.064 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-03-09T08:42:10.088 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:10 +0000 (0:00:00.210) 0:02:30.269 ********* 2021-03-09T08:42:10.117 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:42:10.119 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-03-09T08:42:10.142 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:10 +0000 (0:00:00.053) 0:02:30.322 ********* 2021-03-09T08:42:10.321 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-03-09T08:42:10.344 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:10 +0000 (0:00:00.201) 0:02:30.524 ********* 2021-03-09T08:42:10.405 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T08:42:10.406 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T08:42:10.430 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:10 +0000 (0:00:00.085) 0:02:30.610 ********* 2021-03-09T08:42:10.617 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-09T08:42:10.654 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:10 +0000 (0:00:00.224) 0:02:30.834 ********* 2021-03-09T08:42:10.976 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T08:42:10.978 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-03-09T08:42:11.000 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:10 +0000 (0:00:00.345) 0:02:31.180 ********* 2021-03-09T08:42:11.197 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-03-09T08:42:11.224 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:11 +0000 (0:00:00.223) 0:02:31.404 ********* 2021-03-09T08:42:11.540 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T08:42:11.541 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-03-09T08:42:11.576 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:11 +0000 (0:00:00.349) 0:02:31.754 ********* 2021-03-09T08:42:11.781 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-03-09T08:42:11.814 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:11 +0000 (0:00:00.239) 0:02:31.994 ********* 2021-03-09T08:42:14.143 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T08:42:14.144 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.309479", "end": "2021-03-09 08:42:14.115401", "rc": 0, "start": "2021-03-09 08:42:11.805922", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:14.166 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:14 +0000 (0:00:02.352) 0:02:34.347 ********* 2021-03-09T08:42:14.403 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.359826", "end": "2021-03-09 08:42:14.381781", "rc": 0, "start": "2021-03-09 08:42:12.021955", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:14.429 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:14 +0000 (0:00:00.262) 0:02:34.610 ********* 2021-03-09T08:42:14.702 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-09T08:42:14.703 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-03-09T08:42:14.956 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-03-09T08:42:15.138 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-03-09T08:42:15.443 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-03-09T08:42:15.650 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-03-09T08:42:15.669 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:15 +0000 (0:00:01.238) 0:02:35.849 ********* 2021-03-09T08:42:15.919 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-03-09T08:42:15.945 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:15 +0000 (0:00:00.276) 0:02:36.125 ********* 2021-03-09T08:42:16.195 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-09T08:42:16.197 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-03-09T08:42:16.223 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:16 +0000 (0:00:00.277) 0:02:36.403 ********* 2021-03-09T08:42:16.507 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-03-09T08:42:16.543 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:16 +0000 (0:00:00.320) 0:02:36.723 ********* 2021-03-09T08:42:16.775 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-09T08:42:16.776 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-03-09T08:42:17.073 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-03-09T08:42:17.193 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-03-09T08:42:17.521 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-03-09T08:42:17.637 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-03-09T08:42:17.666 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:17 +0000 (0:00:01.120) 0:02:37.844 ********* 2021-03-09T08:42:17.736 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-09T08:42:17.764 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:17 +0000 (0:00:00.099) 0:02:37.944 ********* 2021-03-09T08:42:17.846 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-09T08:42:17.847 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:17.870 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:17 +0000 (0:00:00.105) 0:02:38.050 ********* 2021-03-09T08:42:17.962 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-09T08:42:17.964 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-03-09T08:42:17.968 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-09T08:42:17.969 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:17.970 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:17.992 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:17 +0000 (0:00:00.122) 0:02:38.173 ********* 2021-03-09T08:42:18.014 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.020) 0:02:38.194 ********* 2021-03-09T08:42:18.065 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:18.067 INFO:teuthology.task.ansible.out:TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:18.092 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.078) 0:02:38.272 ********* 2021-03-09T08:42:18.098 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-09T08:42:18.126 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.034) 0:02:38.306 ********* 2021-03-09T08:42:18.187 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:42:18.189 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-03-09T08:42:18.217 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.091) 0:02:38.397 ********* 2021-03-09T08:42:18.225 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-09T08:42:18.226 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:18.250 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.032) 0:02:38.430 ********* 2021-03-09T08:42:18.289 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-09T08:42:18.291 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:18.314 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.064) 0:02:38.494 ********* 2021-03-09T08:42:18.332 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-09T08:42:18.334 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:18.357 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.043) 0:02:38.537 ********* 2021-03-09T08:42:18.387 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-09T08:42:18.389 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:18.412 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.054) 0:02:38.592 ********* 2021-03-09T08:42:18.441 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-09T08:42:18.442 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:18.466 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.054) 0:02:38.646 ********* 2021-03-09T08:42:18.548 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:42:18.550 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-03-09T08:42:18.577 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.111) 0:02:38.757 ********* 2021-03-09T08:42:18.671 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-09T08:42:18.762 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.117) 0:02:38.875 ********* TASK [testnode : Check if ceph-debuginfo is installed] ************************* fatal: [smithi099.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007105", "end": "2021-03-09 08:42:18.636209", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 08:42:18.629104", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.036) 0:02:38.911 ********* 2021-03-09T08:42:18.865 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-09T08:42:18.868 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:18.895 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.162) 0:02:39.073 ********* 2021-03-09T08:42:18.946 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:18.974 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:18 +0000 (0:00:00.081) 0:02:39.154 ********* 2021-03-09T08:42:19.185 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-09T08:42:19.187 INFO:teuthology.task.ansible.out:fatal: [smithi195.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007191", "end": "2021-03-09 08:42:19.164752", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 08:42:19.157561", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-03-09T08:42:19.211 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:19 +0000 (0:00:00.235) 0:02:39.390 ********* 2021-03-09T08:42:19.302 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:19.304 INFO:teuthology.task.ansible.out:TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:19.344 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:19 +0000 (0:00:00.133) 0:02:39.524 ********* 2021-03-09T08:42:22.167 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T08:42:22.169 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-03-09T08:42:22.192 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:22 +0000 (0:00:02.848) 0:02:42.372 ********* 2021-03-09T08:42:22.615 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-03-09T08:42:22.641 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:22 +0000 (0:00:00.449) 0:02:42.822 ********* 2021-03-09T08:42:24.785 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T08:42:24.787 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-03-09T08:42:24.811 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:24 +0000 (0:00:02.169) 0:02:44.991 ********* 2021-03-09T08:42:25.322 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-03-09T08:42:25.345 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:25 +0000 (0:00:00.533) 0:02:45.525 ********* 2021-03-09T08:42:28.386 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:42:28.388 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-03-09T08:42:28.411 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:28 +0000 (0:00:03.066) 0:02:48.591 ********* 2021-03-09T08:42:28.481 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:28.504 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:28 +0000 (0:00:00.092) 0:02:48.684 ********* 2021-03-09T08:42:29.126 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:42:29.128 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+704+02c27f72.3.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64"]} 2021-03-09T08:42:29.150 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:29 +0000 (0:00:00.646) 0:02:49.331 ********* 2021-03-09T08:42:29.231 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-09T08:42:29.233 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:29.256 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:29 +0000 (0:00:00.105) 0:02:49.437 ********* 2021-03-09T08:42:29.906 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T08:42:29.908 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:42:29.930 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:29 +0000 (0:00:00.673) 0:02:50.111 ********* 2021-03-09T08:42:30.630 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:42:30.664 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:30 +0000 (0:00:00.727) 0:02:50.838 ********* 2021-03-09T08:42:31.322 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:42:31.324 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:42:31.345 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:31 +0000 (0:00:00.687) 0:02:51.525 ********* 2021-03-09T08:42:31.707 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-09T08:42:31.708 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.service system.slice dbus.socket sysinit.target basic.target polkit.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service nftables.service shutdown.target ip6tables.service ipset.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-03-09T08:42:31.710 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-03-09T08:42:31.711 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127252", "LimitNPROCSoft": "127252", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127252", "LimitSIGPENDINGSoft": "127252", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-03-09T08:42:31.713 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:42:31.731 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:31 +0000 (0:00:00.386) 0:02:51.912 ********* 2021-03-09T08:42:32.082 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi099.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-03-09T08:42:32.104 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:32 +0000 (0:00:00.372) 0:02:52.285 ********* 2021-03-09T08:42:32.137 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:42:32.138 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:42:32.170 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:32 +0000 (0:00:00.065) 0:02:52.350 ********* 2021-03-09T08:42:32.576 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-09T08:42:32.577 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket sysinit.target dbus.service system.slice polkit.service basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "nftables.service ip6tables.service ipset.service shutdown.target ebtables.service iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "D 2021-03-09T08:42:32.579 INFO:teuthology.task.ansible.out:evicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "Limi 2021-03-09T08:42:32.581 INFO:teuthology.task.ansible.out:tNOFILESoft": "1024", "LimitNPROC": "127253", "LimitNPROCSoft": "127253", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127253", "LimitSIGPENDINGSoft": "127253", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "S 2021-03-09T08:42:32.584 INFO:teuthology.task.ansible.out:tartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-09T08:42:32.601 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:32 +0000 (0:00:00.431) 0:02:52.781 ********* 2021-03-09T08:42:32.889 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-09T08:42:32.891 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-03-09T08:42:32.915 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:32 +0000 (0:00:00.313) 0:02:53.095 ********* 2021-03-09T08:42:32.955 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-09T08:42:32.957 INFO:teuthology.task.ansible.out:fatal: [smithi195.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-03-09T08:42:32.980 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:32 +0000 (0:00:00.064) 0:02:53.160 ********* 2021-03-09T08:42:32.997 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-09T08:42:32.999 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.027 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.047) 0:02:53.207 ********* 2021-03-09T08:42:33.107 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-09T08:42:33.109 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.130 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.103) 0:02:53.311 ********* 2021-03-09T08:42:33.211 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-09T08:42:33.213 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.237 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.106) 0:02:53.417 ********* 2021-03-09T08:42:33.299 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-03-09T08:42:33.322 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.084) 0:02:53.502 ********* 2021-03-09T08:42:33.322 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:33.330 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:33.333 INFO:teuthology.task.ansible.out:TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.356 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.033) 0:02:53.536 ********* 2021-03-09T08:42:33.357 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:33.396 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-09T08:42:33.397 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.420 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.064) 0:02:53.601 ********* 2021-03-09T08:42:33.442 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-09T08:42:33.443 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.466 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.045) 0:02:53.647 ********* 2021-03-09T08:42:33.517 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:33.519 INFO:teuthology.task.ansible.out:TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.547 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.080) 0:02:53.727 ********* 2021-03-09T08:42:33.570 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-09T08:42:33.573 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.605 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.056) 0:02:53.784 ********* 2021-03-09T08:42:33.644 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-09T08:42:33.646 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.669 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.064) 0:02:53.849 ********* 2021-03-09T08:42:33.687 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-09T08:42:33.739 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.069) 0:02:53.918 ********* 2021-03-09T08:42:33.770 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-09T08:42:33.772 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.810 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.071) 0:02:53.990 ********* 2021-03-09T08:42:33.841 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-09T08:42:33.843 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.871 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.060) 0:02:54.051 ********* 2021-03-09T08:42:33.911 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-09T08:42:33.913 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:33.940 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:33 +0000 (0:00:00.069) 0:02:54.120 ********* 2021-03-09T08:42:33.991 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-09T08:42:33.993 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.025 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.085) 0:02:54.205 ********* 2021-03-09T08:42:34.035 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-09T08:42:34.037 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.063 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.037) 0:02:54.243 ********* 2021-03-09T08:42:34.102 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T08:42:34.104 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.114 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.137 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.074) 0:02:54.318 ********* 2021-03-09T08:42:34.147 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-09T08:42:34.173 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.035) 0:02:54.353 ********* 2021-03-09T08:42:34.234 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:34.236 INFO:teuthology.task.ansible.out:TASK [testnode : Install python-apt] ******************************************* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.261 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.087) 0:02:54.441 ********* 2021-03-09T08:42:34.271 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-09T08:42:34.272 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.300 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.038) 0:02:54.480 ********* 2021-03-09T08:42:34.351 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-09T08:42:34.377 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.076) 0:02:54.557 ********* 2021-03-09T08:42:34.396 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-09T08:42:34.398 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.424 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.047) 0:02:54.605 ********* 2021-03-09T08:42:34.459 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-09T08:42:34.460 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.486 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.060) 0:02:54.665 ********* 2021-03-09T08:42:34.517 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-09T08:42:34.518 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.544 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.058) 0:02:54.724 ********* 2021-03-09T08:42:34.572 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-09T08:42:34.575 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.596 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.052) 0:02:54.776 ********* 2021-03-09T08:42:34.625 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-09T08:42:34.627 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.649 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.053) 0:02:54.829 ********* 2021-03-09T08:42:34.689 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-09T08:42:34.697 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.715 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.065) 0:02:54.895 ********* 2021-03-09T08:42:34.734 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-09T08:42:34.761 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.046) 0:02:54.941 ********* 2021-03-09T08:42:34.801 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.826 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.065) 0:02:55.006 ********* 2021-03-09T08:42:34.845 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-09T08:42:34.847 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.872 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.045) 0:02:55.052 ********* 2021-03-09T08:42:34.906 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.932 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.060) 0:02:55.112 ********* 2021-03-09T08:42:34.950 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-09T08:42:34.951 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:34.974 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:34 +0000 (0:00:00.041) 0:02:55.154 ********* 2021-03-09T08:42:35.003 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:42:35.004 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.030 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.056) 0:02:55.210 ********* 2021-03-09T08:42:35.069 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-09T08:42:35.071 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.093 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.062) 0:02:55.273 ********* 2021-03-09T08:42:35.125 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-09T08:42:35.127 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.149 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.055) 0:02:55.329 ********* 2021-03-09T08:42:35.178 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:35.180 INFO:teuthology.task.ansible.out:TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.210 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.061) 0:02:55.391 ********* 2021-03-09T08:42:35.266 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-09T08:42:35.268 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.292 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.081) 0:02:55.472 ********* 2021-03-09T08:42:35.323 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-09T08:42:35.325 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.350 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.058) 0:02:55.530 ********* 2021-03-09T08:42:35.379 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T08:42:35.381 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.404 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.053) 0:02:55.584 ********* 2021-03-09T08:42:35.499 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-09T08:42:35.501 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.522 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.118) 0:02:55.702 ********* 2021-03-09T08:42:35.575 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:42:35.577 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.602 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.080) 0:02:55.782 ********* 2021-03-09T08:42:35.623 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-09T08:42:35.625 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.647 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.044) 0:02:55.827 ********* 2021-03-09T08:42:35.697 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.726 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.078) 0:02:55.906 ********* 2021-03-09T08:42:35.743 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-09T08:42:35.745 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.767 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.040) 0:02:55.947 ********* 2021-03-09T08:42:35.816 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-09T08:42:35.818 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.842 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.075) 0:02:56.022 ********* 2021-03-09T08:42:35.859 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-09T08:42:35.861 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.883 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.041) 0:02:56.063 ********* 2021-03-09T08:42:35.923 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-09T08:42:35.924 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.959 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.076) 0:02:56.139 ********* 2021-03-09T08:42:35.968 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-09T08:42:35.970 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:35.993 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:35 +0000 (0:00:00.033) 0:02:56.173 ********* 2021-03-09T08:42:36.031 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:36.033 INFO:teuthology.task.ansible.out:TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.056 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.063) 0:02:56.236 ********* 2021-03-09T08:42:36.076 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-09T08:42:36.078 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.100 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.044) 0:02:56.280 ********* 2021-03-09T08:42:36.129 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-09T08:42:36.131 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.153 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.052) 0:02:56.333 ********* 2021-03-09T08:42:36.171 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T08:42:36.173 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.195 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.041) 0:02:56.375 ********* 2021-03-09T08:42:36.245 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-09T08:42:36.247 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.269 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.073) 0:02:56.449 ********* 2021-03-09T08:42:36.287 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:42:36.289 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.311 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.042) 0:02:56.491 ********* 2021-03-09T08:42:36.340 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-09T08:42:36.342 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.365 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.054) 0:02:56.545 ********* 2021-03-09T08:42:36.393 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T08:42:36.395 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.418 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.053) 0:02:56.598 ********* 2021-03-09T08:42:36.437 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-09T08:42:36.439 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.463 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.044) 0:02:56.643 ********* 2021-03-09T08:42:36.513 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:36.515 INFO:teuthology.task.ansible.out:TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.539 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.075) 0:02:56.719 ********* 2021-03-09T08:42:36.560 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-09T08:42:36.562 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.586 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.046) 0:02:56.766 ********* 2021-03-09T08:42:36.830 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T08:42:36.832 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.855 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.268) 0:02:57.035 ********* 2021-03-09T08:42:36.862 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-09T08:42:36.864 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:36.888 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:36 +0000 (0:00:00.033) 0:02:57.068 ********* 2021-03-09T08:42:37.093 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T08:42:37.095 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.122 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.233) 0:02:57.302 ********* 2021-03-09T08:42:37.245 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-09T08:42:37.246 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.270 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.148) 0:02:57.451 ********* 2021-03-09T08:42:37.371 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-09T08:42:37.373 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.408 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.136) 0:02:57.587 ********* 2021-03-09T08:42:37.416 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.444 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.036) 0:02:57.624 ********* 2021-03-09T08:42:37.535 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T08:42:37.536 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.564 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.120) 0:02:57.744 ********* 2021-03-09T08:42:37.584 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-09T08:42:37.587 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.625 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.061) 0:02:57.806 ********* 2021-03-09T08:42:37.654 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:42:37.656 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.683 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.057) 0:02:57.863 ********* 2021-03-09T08:42:37.716 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-09T08:42:37.717 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.740 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.056) 0:02:57.920 ********* 2021-03-09T08:42:37.789 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-09T08:42:37.791 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.813 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.073) 0:02:57.993 ********* 2021-03-09T08:42:37.849 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-09T08:42:37.851 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.879 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.066) 0:02:58.059 ********* 2021-03-09T08:42:37.898 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-09T08:42:37.900 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:37.926 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:37 +0000 (0:00:00.046) 0:02:58.106 ********* 2021-03-09T08:42:37.985 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-09T08:42:37.987 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:38.020 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:38 +0000 (0:00:00.094) 0:02:58.200 ********* 2021-03-09T08:42:38.063 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:42:38.065 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:38.104 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:38 +0000 (0:00:00.082) 0:02:58.283 ********* 2021-03-09T08:42:38.143 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-09T08:42:38.145 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:38.170 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:38 +0000 (0:00:00.067) 0:02:58.350 ********* 2021-03-09T08:42:38.199 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T08:42:38.201 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:38.225 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:38 +0000 (0:00:00.055) 0:02:58.405 ********* 2021-03-09T08:42:38.484 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:38.507 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:38 +0000 (0:00:00.281) 0:02:58.687 ********* 2021-03-09T08:42:38.672 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:42:38.674 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-03-09T08:42:38.706 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:38 +0000 (0:00:00.196) 0:02:58.884 ********* 2021-03-09T08:42:38.784 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:38.819 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:38 +0000 (0:00:00.115) 0:02:58.999 ********* 2021-03-09T08:42:38.820 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:39.054 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:39.077 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:39 +0000 (0:00:00.258) 0:02:59.257 ********* 2021-03-09T08:42:39.141 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:42:39.168 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:39 +0000 (0:00:00.090) 0:02:59.348 ********* 2021-03-09T08:42:39.170 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:39.312 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-09T08:42:39.314 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:39.469 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:39 +0000 (0:00:00.298) 0:02:59.646 ********* 2021-03-09T08:42:39.570 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:39.594 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:39 +0000 (0:00:00.127) 0:02:59.774 ********* 2021-03-09T08:42:39.622 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:42:39.624 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-03-09T08:42:39.654 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:39 +0000 (0:00:00.059) 0:02:59.834 ********* 2021-03-09T08:42:39.770 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T08:42:39.771 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:39.804 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:39 +0000 (0:00:00.150) 0:02:59.985 ********* 2021-03-09T08:42:39.877 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:42:39.879 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-03-09T08:42:39.901 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:39 +0000 (0:00:00.096) 0:03:00.081 ********* 2021-03-09T08:42:39.907 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:42:39.909 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:39.942 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:39 +0000 (0:00:00.041) 0:03:00.122 ********* 2021-03-09T08:42:40.046 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.071 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.128) 0:03:00.251 ********* 2021-03-09T08:42:40.153 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:42:40.155 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:42:40.179 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.108) 0:03:00.360 ********* 2021-03-09T08:42:40.195 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.233 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.053) 0:03:00.413 ********* 2021-03-09T08:42:40.324 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:40.326 INFO:teuthology.task.ansible.out:TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.347 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.113) 0:03:00.527 ********* 2021-03-09T08:42:40.355 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:40.460 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:42:40.462 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.464 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-03-09T08:42:40.501 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.154) 0:03:00.682 ********* 2021-03-09T08:42:40.543 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.041) 0:03:00.723 ********* 2021-03-09T08:42:40.615 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:42:40.617 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.656 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.113) 0:03:00.836 ********* 2021-03-09T08:42:40.670 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T08:42:40.672 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.698 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.041) 0:03:00.878 ********* 2021-03-09T08:42:40.792 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:42:40.794 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.817 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.118) 0:03:00.997 ********* 2021-03-09T08:42:40.836 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:42:40.837 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.866 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.048) 0:03:01.046 ********* 2021-03-09T08:42:40.897 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:42:40.898 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.932 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.066) 0:03:01.112 ********* 2021-03-09T08:42:40.972 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-09T08:42:40.974 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:40.999 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:40 +0000 (0:00:00.066) 0:03:01.179 ********* 2021-03-09T08:42:41.048 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-09T08:42:41.050 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.072 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.073) 0:03:01.252 ********* 2021-03-09T08:42:41.097 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.114 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.042) 0:03:01.294 ********* 2021-03-09T08:42:41.167 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-09T08:42:41.169 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.193 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.078) 0:03:01.373 ********* 2021-03-09T08:42:41.230 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-09T08:42:41.232 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.260 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.066) 0:03:01.440 ********* 2021-03-09T08:42:41.301 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T08:42:41.303 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.327 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.066) 0:03:01.507 ********* 2021-03-09T08:42:41.356 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:42:41.357 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.380 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.053) 0:03:01.560 ********* 2021-03-09T08:42:41.411 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:42:41.413 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.435 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.055) 0:03:01.616 ********* 2021-03-09T08:42:41.464 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-09T08:42:41.466 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.497 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.061) 0:03:01.677 ********* 2021-03-09T08:42:41.517 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-09T08:42:41.518 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.544 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.046) 0:03:01.724 ********* 2021-03-09T08:42:41.616 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.654 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.109) 0:03:01.834 ********* 2021-03-09T08:42:41.665 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-09T08:42:41.667 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.703 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.048) 0:03:01.883 ********* 2021-03-09T08:42:41.726 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-09T08:42:41.728 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.756 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.053) 0:03:01.936 ********* 2021-03-09T08:42:41.795 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-09T08:42:41.797 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.824 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.067) 0:03:02.004 ********* 2021-03-09T08:42:41.842 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-09T08:42:41.844 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.868 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.044) 0:03:02.048 ********* 2021-03-09T08:42:41.922 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-09T08:42:41.924 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:41.947 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.078) 0:03:02.127 ********* 2021-03-09T08:42:41.975 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-09T08:42:41.977 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.000 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:41 +0000 (0:00:00.053) 0:03:02.181 ********* 2021-03-09T08:42:42.051 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.075 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.074) 0:03:02.255 ********* 2021-03-09T08:42:42.128 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-09T08:42:42.129 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.155 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.080) 0:03:02.335 ********* 2021-03-09T08:42:42.172 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-09T08:42:42.174 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.197 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.041) 0:03:02.377 ********* 2021-03-09T08:42:42.235 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-09T08:42:42.237 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.263 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.066) 0:03:02.443 ********* 2021-03-09T08:42:42.281 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-09T08:42:42.282 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.306 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.042) 0:03:02.486 ********* 2021-03-09T08:42:42.308 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:42.345 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-09T08:42:42.347 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.385 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.079) 0:03:02.565 ********* 2021-03-09T08:42:42.398 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-09T08:42:42.399 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.427 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.041) 0:03:02.607 ********* 2021-03-09T08:42:42.462 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-09T08:42:42.464 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.493 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.066) 0:03:02.673 ********* 2021-03-09T08:42:42.514 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-09T08:42:42.516 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.541 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.047) 0:03:02.721 ********* 2021-03-09T08:42:42.611 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-09T08:42:42.613 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.636 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.095) 0:03:02.816 ********* 2021-03-09T08:42:42.655 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:42:42.657 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T08:42:42.679 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.043) 0:03:02.859 ********* 2021-03-09T08:42:42.785 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-09T08:42:42.787 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.809 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.129) 0:03:02.989 ********* 2021-03-09T08:42:42.827 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-09T08:42:42.829 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} ok: [smithi099.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi099.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.829 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.830 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.841 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.843 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.882 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.073) 0:03:03.063 ********* 2021-03-09T08:42:42.899 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-09T08:42:42.901 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:42.926 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:42 +0000 (0:00:00.043) 0:03:03.106 ********* 2021-03-09T08:42:43.043 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.065 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.139) 0:03:03.245 ********* 2021-03-09T08:42:43.136 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:43.138 INFO:teuthology.task.ansible.out:TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.162 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.096) 0:03:03.342 ********* 2021-03-09T08:42:43.211 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.015156", "end": "2021-03-09 08:42:43.183747", "rc": 0, "start": "2021-03-09 08:42:43.168591", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-09T08:42:43.247 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.083) 0:03:03.425 ********* 2021-03-09T08:42:43.253 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-09T08:42:43.254 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.279 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.033) 0:03:03.459 ********* 2021-03-09T08:42:43.281 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:43.330 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-09T08:42:43.332 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.357 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.077) 0:03:03.537 ********* 2021-03-09T08:42:43.375 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-09T08:42:43.377 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.400 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.042) 0:03:03.580 ********* 2021-03-09T08:42:43.449 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-09T08:42:43.451 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.473 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.073) 0:03:03.653 ********* 2021-03-09T08:42:43.492 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:42:43.494 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-09T08:42:43.517 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.044) 0:03:03.698 ********* 2021-03-09T08:42:43.556 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-09T08:42:43.558 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.584 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.066) 0:03:03.764 ********* 2021-03-09T08:42:43.613 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-09T08:42:43.615 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.624 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-09T08:42:43.636 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.637 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.647 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.649 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.658 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.660 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:43.685 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.101) 0:03:03.865 ********* 2021-03-09T08:42:43.708 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:43 +0000 (0:00:00.022) 0:03:03.888 ********* 2021-03-09T08:42:43.984 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004093", "end": "2021-03-09 08:42:43.961120", "rc": 0, "start": "2021-03-09 08:42:43.957027", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-09T08:42:44.009 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:44 +0000 (0:00:00.300) 0:03:04.189 ********* 2021-03-09T08:42:44.089 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-09T08:42:44.091 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:44.115 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:44 +0000 (0:00:00.106) 0:03:04.295 ********* 2021-03-09T08:42:44.206 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-09T08:42:44.208 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:44.230 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:44 +0000 (0:00:00.112) 0:03:04.408 ********* 2021-03-09T08:42:44.297 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:44.299 INFO:teuthology.task.ansible.out:TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:44.322 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:44 +0000 (0:00:00.093) 0:03:04.502 ********* 2021-03-09T08:42:44.403 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-09T08:42:44.404 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:44.428 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:44 +0000 (0:00:00.106) 0:03:04.608 ********* 2021-03-09T08:42:45.054 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-09T08:42:45.056 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:42:45.080 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:45 +0000 (0:00:00.651) 0:03:05.260 ********* 2021-03-09T08:42:45.172 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:42:45.173 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930279505920, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227119020, 'block_used': 12976088, 'inode_total': 61054976, 'inode_available': 60985794, 'inode_used': 69182, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227119020, "block_size": 4096, "block_total": 240095108, "block_used": 12976088, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985794, "inode_total": 61054976, "inode_used": 69182, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930279505920, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}} 2021-03-09T08:42:45.196 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:45 +0000 (0:00:00.116) 0:03:05.376 ********* 2021-03-09T08:42:45.433 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-09T08:42:45.435 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005410", "end": "2021-03-09 08:42:45.409453", "rc": 0, "start": "2021-03-09 08:42:45.404043", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:45.456 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:45 +0000 (0:00:00.260) 0:03:05.637 ********* 2021-03-09T08:42:45.547 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-09T08:42:45.549 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930279505920, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227119020, 'block_used': 12976088, 'inode_total': 61054976, 'inode_available': 60985794, 'inode_used': 69182, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227119020, "block_size": 4096, "block_total": 240095108, "block_used": 12976088, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985794, "inode_total": 61054976, "inode_used": 69182, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930279505920, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:45.571 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:45 +0000 (0:00:00.114) 0:03:05.751 ********* 2021-03-09T08:42:45.642 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-09T08:42:45.666 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:45 +0000 (0:00:00.095) 0:03:05.847 ********* 2021-03-09T08:42:45.821 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-09T08:42:45.823 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:42:45.846 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:45 +0000 (0:00:00.179) 0:03:06.026 ********* 2021-03-09T08:42:45.927 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:42:45.929 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930279518208, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227119023, 'block_used': 12976085, 'inode_total': 61054976, 'inode_available': 60985795, 'inode_used': 69181, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227119023, "block_size": 4096, "block_total": 240095108, "block_used": 12976085, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985795, "inode_total": 61054976, "inode_used": 69181, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930279518208, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}} 2021-03-09T08:42:45.950 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:45 +0000 (0:00:00.104) 0:03:06.130 ********* 2021-03-09T08:42:46.187 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-09T08:42:46.189 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005074", "end": "2021-03-09 08:42:46.163574", "rc": 0, "start": "2021-03-09 08:42:46.158500", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:46.210 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:46 +0000 (0:00:00.259) 0:03:06.390 ********* 2021-03-09T08:42:46.300 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-09T08:42:46.417 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930279518208, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227119023, 'block_used': 12976085, 'inode_total': 61054976, 'inode_available': 60985795, 'inode_used': 69181, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227119023, "block_size": 4096, "block_total": 240095108, "block_used": 12976085, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985795, "inode_total": 61054976, "inode_used": 69181, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930279518208, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} Tuesday 09 March 2021 08:42:46 +0000 (0:00:00.115) 0:03:06.505 ********* TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-09T08:42:46.430 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:46 +0000 (0:00:00.104) 0:03:06.611 ********* 2021-03-09T08:42:47.170 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-09T08:42:47.172 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfuTm7SBERY5rtH4PyZ5Tgg0Z3LH60gVio'], 'uuids': ['2e685f5c-4dfd-4325-ad35-54570861adfd'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.239743", "end": "2021-03-09 08:42:47.127494", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfuTm7SBERY5rtH4PyZ5Tgg0Z3LH60gVio"], "labels": [], "masters": [], "uuids": ["2e685f5c-4dfd-4325-ad35-54570861adfd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:45.887751", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:48.081 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRDTrdi5w5MxHSTdZ88k4OI58STtcpa5cx'], 'uuids': ['ced4dd94-598c-4463-9499-f7882a4dae89'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.391998", "end": "2021-03-09 08:42:48.049662", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRDTrdi5w5MxHSTdZ88k4OI58STtcpa5cx"], "labels": [], "masters": [], "uuids": ["ced4dd94-598c-4463-9499-f7882a4dae89"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:46.657664", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:48.415 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-WLleWG-iRVJ-Nrr9-nS8R-E1q6-nNqk-gQWrnE', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040101400BGN', 'nvme-nvme.8086-50484654363230343031303134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.072670", "end": "2021-03-09 08:42:48.385180", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-WLleWG-iRVJ-Nrr9-nS8R-E1q6-nNqk-gQWrnE", "nvme-INTEL_SSDPEDMD400G4_PHFT62040101400BGN", "nvme-nvme.8086-50484654363230343031303134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:47.312510", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-09T08:42:48.417 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:49.354 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-9otd64-PvMd-oxP8-TLB0-wEEL-0JAr-SK6hge', 'nvme-INTEL_SSDPED1D480GA_PHMB751300FS480DGN', 'nvme-nvme.8086-50484d42373531333030465334383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.092695", "end": "2021-03-09 08:42:49.324528", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-9otd64-PvMd-oxP8-TLB0-wEEL-0JAr-SK6hge", "nvme-INTEL_SSDPED1D480GA_PHMB751300FS480DGN", "nvme-nvme.8086-50484d42373531333030465334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:48.231833", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-03-09T08:42:49.356 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:49.660 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfoJIlIVzao4yoRwIZmQddPZ2l12JOTe59'], 'uuids': ['387e64eb-f74d-4e0e-a91e-13c1608d3500'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.056418", "end": "2021-03-09 08:42:49.627531", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfoJIlIVzao4yoRwIZmQddPZ2l12JOTe59"], "labels": [], "masters": [], "uuids": ["387e64eb-f74d-4e0e-a91e-13c1608d3500"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:48.571113", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:50.601 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRibQaKq5TP4scugShTg8tp8p8dzmIre3o'], 'uuids': ['c8be4bba-319a-4399-b72c-89de772f1214'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.053527", "end": "2021-03-09 08:42:50.574784", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRibQaKq5TP4scugShTg8tp8p8dzmIre3o"], "labels": [], "masters": [], "uuids": ["c8be4bba-319a-4399-b72c-89de772f1214"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:49.521257", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:50.897 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmf9f2pDxFIP1jKeEtEA3xqbO8G8kO83ipj'], 'uuids': ['d29c0e16-c577-4889-8118-3a77e8502621'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.050348", "end": "2021-03-09 08:42:50.867842", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmf9f2pDxFIP1jKeEtEA3xqbO8G8kO83ipj"], "labels": [], "masters": [], "uuids": ["d29c0e16-c577-4889-8118-3a77e8502621"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:49.817494", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:51.881 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRkr1BDdiMxjSMkNkVEOk3XLxQ1JvQVolJ'], 'uuids': ['b6b71356-3065-48a9-8d94-6de0b894aaaf'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.092447", "end": "2021-03-09 08:42:51.848448", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRkr1BDdiMxjSMkNkVEOk3XLxQ1JvQVolJ"], "labels": [], "masters": [], "uuids": ["b6b71356-3065-48a9-8d94-6de0b894aaaf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:50.756001", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:52.125 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfC2RkYSDKR8fptA3lx0j4isZlQRHMd2m5'], 'uuids': ['113bf6fa-a95f-44bf-bbfa-e446158a1d0d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.044382", "end": "2021-03-09 08:42:52.093791", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfC2RkYSDKR8fptA3lx0j4isZlQRHMd2m5"], "labels": [], "masters": [], "uuids": ["113bf6fa-a95f-44bf-bbfa-e446158a1d0d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:51.049409", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:52.127 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LHPN', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LHPN', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LHPN', 'scsi-35000c500918da8bc', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LHPN', 'wwn-0x5000c500918da8bc'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5LHPN', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918da8bc', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LHPN-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LHPN-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LHPN-part1', 'scsi-35000c500918da8bc-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LHPN-part1', 'wwn-0x5000c500918da8bc-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LHPN", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LHPN", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LHPN", "scsi-35000c500918da8bc", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LHPN", "wwn-0x5000c500918da8bc"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LHPN-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LHPN-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-03-09T08:42:52.129 INFO:teuthology.task.ansible.out:Z1W5LHPN-part1", "scsi-35000c500918da8bc-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LHPN-part1", "wwn-0x5000c500918da8bc-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LHPN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918da8bc"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:53.137 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRSYPbafH6YZEjut9qwE5uMIcjN5fxnRJ2'], 'uuids': ['ba4df4fe-e252-42b6-8377-4689956ad645'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.071043", "end": "2021-03-09 08:42:53.100806", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRSYPbafH6YZEjut9qwE5uMIcjN5fxnRJ2"], "labels": [], "masters": [], "uuids": ["ba4df4fe-e252-42b6-8377-4689956ad645"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:52.029763", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:53.143 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLZZ', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLZZ', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLZZ', 'scsi-35000c50092934caa', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XLZZ', 'wwn-0x5000c50092934caa'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XLZZ', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092934caa', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLZZ-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLZZ-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLZZ-part1', 'scsi-35000c50092934caa-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XLZZ-part1', 'wwn-0x5000c50092934caa-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLZZ", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLZZ", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLZZ", "scsi-35000c50092934caa", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XLZZ", "wwn-0x5000c50092934caa"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLZZ-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLZZ-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-03-09T08:42:53.145 INFO:teuthology.task.ansible.out:Z1W5XLZZ-part1", "scsi-35000c50092934caa-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XLZZ-part1", "wwn-0x5000c50092934caa-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLZZ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092934caa"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:53.371 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmf8Y97aWHAKbHz0JdBthzPl4M3rKZY0ofj'], 'uuids': ['0eb5e9d9-4dd9-4466-91ed-759970ce83a6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.055659", "end": "2021-03-09 08:42:53.336143", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmf8Y97aWHAKbHz0JdBthzPl4M3rKZY0ofj"], "labels": [], "masters": [], "uuids": ["0eb5e9d9-4dd9-4466-91ed-759970ce83a6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:52.280484", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:53.400 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:53 +0000 (0:00:06.969) 0:03:13.580 ********* 2021-03-09T08:42:53.822 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:42:53.824 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfuTm7SBERY5rtH4PyZ5Tgg0Z3LH60gVio'], 'uuids': ['2e685f5c-4dfd-4325-ad35-54570861adfd'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.140952", "end": "2021-03-09 08:42:53.788166", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfuTm7SBERY5rtH4PyZ5Tgg0Z3LH60gVio"], "labels": [], "masters": [], "uuids": ["2e685f5c-4dfd-4325-ad35-54570861adfd"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:53.647214", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138249 s, 834 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138249 s, 834 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:54.027 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-WLleWG-iRVJ-Nrr9-nS8R-E1q6-nNqk-gQWrnE', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040101400BGN', 'nvme-nvme.8086-50484654363230343031303134303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037695", "end": "2021-03-09 08:42:53.999590", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-WLleWG-iRVJ-Nrr9-nS8R-E1q6-nNqk-gQWrnE", "nvme-INTEL_SSDPEDMD400G4_PHFT62040101400BGN", "nvme-nvme.8086-50484654363230343031303134303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:53.961895", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0351133 s, 3.3 GB/s", "std 2021-03-09T08:42:54.028 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0351133 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:54.349 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfoJIlIVzao4yoRwIZmQddPZ2l12JOTe59'], 'uuids': ['387e64eb-f74d-4e0e-a91e-13c1608d3500'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.152442", "end": "2021-03-09 08:42:54.325880", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfoJIlIVzao4yoRwIZmQddPZ2l12JOTe59"], "labels": [], "masters": [], "uuids": ["387e64eb-f74d-4e0e-a91e-13c1608d3500"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:54.173438", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149868 s, 770 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149868 s, 770 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:54.395 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRkPmBcFLRkjId6w8NK7eQHQhysxYIT9Rv'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.065042", "end": "2021-03-09 08:42:54.359405", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRkPmBcFLRkjId6w8NK7eQHQhysxYIT9Rv"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:53.294363", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-09T08:42:54.431 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:54 +0000 (0:00:01.031) 0:03:14.611 ********* 2021-03-09T08:42:54.669 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:42:54.670 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmf9f2pDxFIP1jKeEtEA3xqbO8G8kO83ipj'], 'uuids': ['d29c0e16-c577-4889-8118-3a77e8502621'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.144512", "end": "2021-03-09 08:42:54.643682", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmf9f2pDxFIP1jKeEtEA3xqbO8G8kO83ipj"], "labels": [], "masters": [], "uuids": ["d29c0e16-c577-4889-8118-3a77e8502621"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:54.499170", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14192 s, 813 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14192 s, 813 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:54.792 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRDTrdi5w5MxHSTdZ88k4OI58STtcpa5cx'], 'uuids': ['ced4dd94-598c-4463-9499-f7882a4dae89'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.095905", "end": "2021-03-09 08:42:54.760960", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRDTrdi5w5MxHSTdZ88k4OI58STtcpa5cx"], "labels": [], "masters": [], "uuids": ["ced4dd94-598c-4463-9499-f7882a4dae89"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:54.665055", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0932789 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0932789 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:54.982 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfC2RkYSDKR8fptA3lx0j4isZlQRHMd2m5'], 'uuids': ['113bf6fa-a95f-44bf-bbfa-e446158a1d0d'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.145707", "end": "2021-03-09 08:42:54.958023", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmfC2RkYSDKR8fptA3lx0j4isZlQRHMd2m5"], "labels": [], "masters": [], "uuids": ["113bf6fa-a95f-44bf-bbfa-e446158a1d0d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:54.812316", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14307 s, 806 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14307 s, 806 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:54.994 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LHPN', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LHPN', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LHPN', 'scsi-35000c500918da8bc', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LHPN', 'wwn-0x5000c500918da8bc'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5LHPN', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918da8bc', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LHPN-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LHPN-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LHPN-part1', 'scsi-35000c500918da8bc-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LHPN-part1', 'wwn-0x5000c500918da8bc-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LHPN", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LHPN", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LHPN", "scsi-35000c500918da8bc", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LHPN", "wwn-0x5000c500918da8bc"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LHPN-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LHPN-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-03-09T08:42:54.996 INFO:teuthology.task.ansible.out:Z1W5LHPN-part1", "scsi-35000c500918da8bc-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LHPN-part1", "wwn-0x5000c500918da8bc-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LHPN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918da8bc"}}, "skip_reason": "Conditional result was False"} changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-9otd64-PvMd-oxP8-TLB0-wEEL-0JAr-SK6hge', 'nvme-INTEL_SSDPED1D480GA_PHMB751300FS480DGN', 'nvme-nvme.8086-50484d42373531333030465334383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.038033", "end": "2021-03-09 08:42:54.970812", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-9otd64-PvMd-oxP8-TLB0-wEEL-0JAr-SK6hge", "nvme-INTEL_SSDP 2021-03-09T08:42:54.998 INFO:teuthology.task.ansible.out:ED1D480GA_PHMB751300FS480DGN", "nvme-nvme.8086-50484d42373531333030465334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:54.932779", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0353483 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0353483 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:55.258 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRibQaKq5TP4scugShTg8tp8p8dzmIre3o'], 'uuids': ['c8be4bba-319a-4399-b72c-89de772f1214'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.096525", "end": "2021-03-09 08:42:55.237262", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRibQaKq5TP4scugShTg8tp8p8dzmIre3o"], "labels": [], "masters": [], "uuids": ["c8be4bba-319a-4399-b72c-89de772f1214"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:55.140737", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0938928 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0938928 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:55.302 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmf8Y97aWHAKbHz0JdBthzPl4M3rKZY0ofj'], 'uuids': ['0eb5e9d9-4dd9-4466-91ed-759970ce83a6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.142554", "end": "2021-03-09 08:42:55.277985", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-PX5mzUulYycnHAHQekiiukb0U8kt1bmf8Y97aWHAKbHz0JdBthzPl4M3rKZY0ofj"], "labels": [], "masters": [], "uuids": ["0eb5e9d9-4dd9-4466-91ed-759970ce83a6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:55.135431", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139953 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139953 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:55.335 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:55 +0000 (0:00:00.903) 0:03:15.515 ********* 2021-03-09T08:42:55.532 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRkr1BDdiMxjSMkNkVEOk3XLxQ1JvQVolJ'], 'uuids': ['b6b71356-3065-48a9-8d94-6de0b894aaaf'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.095664", "end": "2021-03-09 08:42:55.502179", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRkr1BDdiMxjSMkNkVEOk3XLxQ1JvQVolJ"], "labels": [], "masters": [], "uuids": ["b6b71356-3065-48a9-8d94-6de0b894aaaf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:55.406515", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.093107 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.093107 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:55.617 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-09T08:42:55.619 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.055553", "end": "2021-03-09 08:42:55.547741", "rc": 0, "start": "2021-03-09 08:42:55.492188", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:55.640 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:55 +0000 (0:00:00.304) 0:03:15.820 ********* 2021-03-09T08:42:55.795 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:42:55.797 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRSYPbafH6YZEjut9qwE5uMIcjN5fxnRJ2'], 'uuids': ['ba4df4fe-e252-42b6-8377-4689956ad645'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.096975", "end": "2021-03-09 08:42:55.768521", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRSYPbafH6YZEjut9qwE5uMIcjN5fxnRJ2"], "labels": [], "masters": [], "uuids": ["ba4df4fe-e252-42b6-8377-4689956ad645"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:55.671546", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0943529 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0943529 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:55.808 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLZZ', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLZZ', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLZZ', 'scsi-35000c50092934caa', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XLZZ', 'wwn-0x5000c50092934caa'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XLZZ', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50092934caa', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLZZ-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLZZ-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLZZ-part1', 'scsi-35000c50092934caa-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XLZZ-part1', 'wwn-0x5000c50092934caa-part1'], 'uuids': ['a9bc468f-7d76-441a-b17b-cd059807719c'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLZZ", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLZZ", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLZZ", "scsi-35000c50092934caa", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XLZZ", "wwn-0x5000c50092934caa"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLZZ-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLZZ-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-03-09T08:42:55.810 INFO:teuthology.task.ansible.out:Z1W5XLZZ-part1", "scsi-35000c50092934caa-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XLZZ-part1", "wwn-0x5000c50092934caa-part1"], "labels": [], "masters": [], "uuids": ["a9bc468f-7d76-441a-b17b-cd059807719c"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLZZ", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50092934caa"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:55.862 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.043677", "end": "2021-03-09 08:42:55.843553", "rc": 0, "start": "2021-03-09 08:42:55.799876", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:55.885 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:55 +0000 (0:00:00.245) 0:03:16.065 ********* 2021-03-09T08:42:55.955 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-09T08:42:56.015 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.129) 0:03:16.195 ********* 2021-03-09T08:42:56.042 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-09T08:42:56.044 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.072 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.057) 0:03:16.253 ********* 2021-03-09T08:42:56.080 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-09T08:42:56.081 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRkPmBcFLRkjId6w8NK7eQHQhysxYIT9Rv'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.095332", "end": "2021-03-09 08:42:56.039273", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-6cWlg4Cam7BOk8bWZUGcaKd7t0uNc5eRkPmBcFLRkjId6w8NK7eQHQhysxYIT9Rv"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-09 08:42:55.943941", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0927424 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0927424 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:56.115 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.042) 0:03:16.295 ********* 2021-03-09T08:42:56.123 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-09T08:42:56.126 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.150 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.034) 0:03:16.330 ********* 2021-03-09T08:42:56.178 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:42:56.180 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.204 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.054) 0:03:16.384 ********* 2021-03-09T08:42:56.233 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-09T08:42:56.235 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.256 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.052) 0:03:16.436 ********* 2021-03-09T08:42:56.368 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-09T08:42:56.370 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.058077", "end": "2021-03-09 08:42:56.339459", "rc": 0, "start": "2021-03-09 08:42:56.281382", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:56.394 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.137) 0:03:16.574 ********* 2021-03-09T08:42:56.445 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-09T08:42:56.446 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:42:56.470 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.076) 0:03:16.650 ********* 2021-03-09T08:42:56.509 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:42:56.511 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930279505920, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227119020, 'block_used': 12976088, 'inode_total': 61054976, 'inode_available': 60985794, 'inode_used': 69182, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227119020, "block_size": 4096, "block_total": 240095108, "block_used": 12976088, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985794, "inode_total": 61054976, "inode_used": 69182, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930279505920, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.535 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.065) 0:03:16.716 ********* 2021-03-09T08:42:56.563 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-09T08:42:56.564 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.586 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.050) 0:03:16.766 ********* 2021-03-09T08:42:56.615 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:42:56.617 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for physical volumes] *********************************** changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.045702", "end": "2021-03-09 08:42:56.596339", "rc": 0, "start": "2021-03-09 08:42:56.550637", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:42:56.638 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.051) 0:03:16.818 ********* 2021-03-09T08:42:56.659 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.020) 0:03:16.839 ********* 2021-03-09T08:42:56.676 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:42:56.678 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.699 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.040) 0:03:16.879 ********* 2021-03-09T08:42:56.706 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-09T08:42:56.734 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.035) 0:03:16.914 ********* 2021-03-09T08:42:56.741 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-09T08:42:56.742 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.764 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.030) 0:03:16.944 ********* 2021-03-09T08:42:56.771 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-09T08:42:56.773 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.800 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.035) 0:03:16.980 ********* 2021-03-09T08:42:56.805 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-09T08:42:56.807 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.829 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.029) 0:03:17.009 ********* 2021-03-09T08:42:56.835 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-09T08:42:56.837 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.859 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.030) 0:03:17.040 ********* 2021-03-09T08:42:56.867 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-09T08:42:56.873 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.895 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.035) 0:03:17.076 ********* 2021-03-09T08:42:56.902 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:42:56.904 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.929 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.033) 0:03:17.109 ********* 2021-03-09T08:42:56.957 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-09T08:42:56.959 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:56.980 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:56 +0000 (0:00:00.051) 0:03:17.160 ********* 2021-03-09T08:42:57.177 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-09T08:42:57.179 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-09T08:42:57.201 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:57 +0000 (0:00:00.220) 0:03:17.381 ********* 2021-03-09T08:42:57.240 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-09T08:42:57.241 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930279518208, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227119023, 'block_used': 12976085, 'inode_total': 61054976, 'inode_available': 60985795, 'inode_used': 69181, 'uuid': 'a9bc468f-7d76-441a-b17b-cd059807719c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227119023, "block_size": 4096, "block_total": 240095108, "block_used": 12976085, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985795, "inode_total": 61054976, "inode_used": 69181, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930279518208, "size_total": 983429562368, "uuid": "a9bc468f-7d76-441a-b17b-cd059807719c"}, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:57.264 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:57 +0000 (0:00:00.063) 0:03:17.444 ********* 2021-03-09T08:42:57.303 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-09T08:42:57.305 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:57.326 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:57 +0000 (0:00:00.061) 0:03:17.506 ********* 2021-03-09T08:42:57.355 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:57.356 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:57.379 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:57 +0000 (0:00:00.053) 0:03:17.559 ********* 2021-03-09T08:42:57.408 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:42:57.409 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:57.432 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:57 +0000 (0:00:00.053) 0:03:17.612 ********* 2021-03-09T08:42:57.460 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-09T08:42:57.462 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:57.482 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:57 +0000 (0:00:00.050) 0:03:17.663 ********* 2021-03-09T08:42:57.489 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-09T08:42:57.490 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T08:42:57.514 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:57 +0000 (0:00:00.031) 0:03:17.694 ********* 2021-03-09T08:42:57.521 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-09T08:42:57.522 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:57.546 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:57 +0000 (0:00:00.031) 0:03:17.726 ********* 2021-03-09T08:42:57.574 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-09T08:42:57.576 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:42:57.597 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:57 +0000 (0:00:00.051) 0:03:17.777 ********* 2021-03-09T08:42:58.025 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-09T08:42:58.027 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-09T08:42:58.049 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:58 +0000 (0:00:00.451) 0:03:18.229 ********* 2021-03-09T08:42:58.139 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T08:42:58.141 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:58.409 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:58.473 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:58.475 INFO:teuthology.task.ansible.out: 2021-03-09T08:42:58.755 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:58.788 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:59.090 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:59.112 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:59.403 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:59.447 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:59.487 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:59 +0000 (0:00:01.437) 0:03:19.667 ********* 2021-03-09T08:42:59.682 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:42:59.684 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T08:42:59.717 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:59 +0000 (0:00:00.230) 0:03:19.897 ********* 2021-03-09T08:42:59.739 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-09T08:42:59.741 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-09T08:42:59.770 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:59 +0000 (0:00:00.052) 0:03:19.950 ********* 2021-03-09T08:42:59.926 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T08:42:59.928 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:42:59.959 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-09T08:42:59.960 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-09T08:42:59.982 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:42:59 +0000 (0:00:00.212) 0:03:20.162 ********* 2021-03-09T08:43:00.094 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T08:43:00.096 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:43:00.190 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:43:00.265 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:43:00.412 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:43:00.443 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:43:00.454 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:00.482 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:00 +0000 (0:00:00.499) 0:03:20.662 ********* 2021-03-09T08:43:00.510 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-09T08:43:00.513 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:00.532 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:00 +0000 (0:00:00.050) 0:03:20.712 ********* 2021-03-09T08:43:00.561 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:00.591 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:00 +0000 (0:00:00.058) 0:03:20.771 ********* 2021-03-09T08:43:00.606 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-09T08:43:00.608 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:43:00.768 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-09T08:43:00.781 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:00.809 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:00 +0000 (0:00:00.217) 0:03:20.989 ********* 2021-03-09T08:43:00.817 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-09T08:43:00.819 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:43:00.842 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:00 +0000 (0:00:00.032) 0:03:21.022 ********* 2021-03-09T08:43:00.848 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-09T08:43:00.850 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:00.873 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:00 +0000 (0:00:00.031) 0:03:21.053 ********* 2021-03-09T08:43:00.880 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-09T08:43:00.883 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T08:43:00.914 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:00 +0000 (0:00:00.041) 0:03:21.094 ********* 2021-03-09T08:43:00.923 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-09T08:43:00.925 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:00.948 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:00 +0000 (0:00:00.034) 0:03:21.129 ********* 2021-03-09T08:43:01.150 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-09T08:43:01.152 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:43:01.177 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:01 +0000 (0:00:00.228) 0:03:21.358 ********* 2021-03-09T08:43:01.218 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-09T08:43:01.220 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-09T08:43:01.242 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:01 +0000 (0:00:00.065) 0:03:21.423 ********* 2021-03-09T08:43:01.482 INFO:teuthology.task.ansible.out: 2021-03-09T08:43:01.484 INFO:teuthology.task.ansible.out:TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi195.front.sepia.ceph.com] => {"changed": true} 2021-03-09T08:43:01.517 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:01 +0000 (0:00:00.274) 0:03:21.697 ********* 2021-03-09T08:43:01.537 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true} 2021-03-09T08:43:01.561 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:01 +0000 (0:00:00.044) 0:03:21.741 ********* 2021-03-09T08:43:02.095 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-09T08:43:02.097 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T08:43:02.120 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.559) 0:03:22.301 ********* 2021-03-09T08:43:02.132 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-09T08:43:02.166 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.045) 0:03:22.346 ********* 2021-03-09T08:43:02.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-09T08:43:02.179 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.201 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.035) 0:03:22.382 ********* 2021-03-09T08:43:02.209 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.236 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.034) 0:03:22.417 ********* 2021-03-09T08:43:02.244 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-09T08:43:02.246 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.267 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.030) 0:03:22.447 ********* 2021-03-09T08:43:02.274 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.298 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.030) 0:03:22.478 ********* 2021-03-09T08:43:02.305 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T08:43:02.306 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.328 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.030) 0:03:22.508 ********* 2021-03-09T08:43:02.334 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.362 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.033) 0:03:22.542 ********* 2021-03-09T08:43:02.369 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-09T08:43:02.370 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.395 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.033) 0:03:22.575 ********* 2021-03-09T08:43:02.403 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.432 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.037) 0:03:22.613 ********* 2021-03-09T08:43:02.441 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-09T08:43:02.443 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.469 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.034) 0:03:22.647 ********* 2021-03-09T08:43:02.474 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.500 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.033) 0:03:22.681 ********* 2021-03-09T08:43:02.511 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-09T08:43:02.512 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.536 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.035) 0:03:22.716 ********* 2021-03-09T08:43:02.543 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.566 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.030) 0:03:22.747 ********* 2021-03-09T08:43:02.573 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-09T08:43:02.575 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.599 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.032) 0:03:22.779 ********* 2021-03-09T08:43:02.607 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.630 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.031) 0:03:22.811 ********* 2021-03-09T08:43:02.637 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-09T08:43:02.639 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.661 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.030) 0:03:22.841 ********* 2021-03-09T08:43:02.668 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.693 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.031) 0:03:22.873 ********* 2021-03-09T08:43:02.700 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-09T08:43:02.702 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.728 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.034) 0:03:22.908 ********* 2021-03-09T08:43:02.735 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.760 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.032) 0:03:22.940 ********* 2021-03-09T08:43:02.767 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-09T08:43:02.769 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.793 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.032) 0:03:22.973 ********* 2021-03-09T08:43:02.800 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.825 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.032) 0:03:23.006 ********* 2021-03-09T08:43:02.827 INFO:teuthology.task.ansible.out: 2021-03-09T08:43:02.835 INFO:teuthology.task.ansible.out: 2021-03-09T08:43:02.836 INFO:teuthology.task.ansible.out:TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.859 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.033) 0:03:23.039 ********* 2021-03-09T08:43:02.862 INFO:teuthology.task.ansible.out: 2021-03-09T08:43:02.869 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.893 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.033) 0:03:23.073 ********* 2021-03-09T08:43:02.900 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-09T08:43:02.901 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.929 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.036) 0:03:23.109 ********* 2021-03-09T08:43:02.936 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.960 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.030) 0:03:23.140 ********* 2021-03-09T08:43:02.967 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-09T08:43:02.968 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:02.989 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:02 +0000 (0:00:00.029) 0:03:23.170 ********* 2021-03-09T08:43:02.996 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:03.020 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:03 +0000 (0:00:00.030) 0:03:23.200 ********* 2021-03-09T08:43:03.028 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-09T08:43:03.030 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:03.061 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:03 +0000 (0:00:00.040) 0:03:23.241 ********* 2021-03-09T08:43:03.067 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:03.093 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:03 +0000 (0:00:00.032) 0:03:23.273 ********* 2021-03-09T08:43:03.102 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-09T08:43:03.104 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:03.126 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:03 +0000 (0:00:00.032) 0:03:23.306 ********* 2021-03-09T08:43:03.133 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:03.164 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:03 +0000 (0:00:00.038) 0:03:23.344 ********* 2021-03-09T08:43:03.617 INFO:teuthology.task.ansible.out: 2021-03-09T08:43:03.623 INFO:teuthology.task.ansible.out:TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-09T08:43:03.624 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-03-09T08:43:03.659 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:03 +0000 (0:00:00.492) 0:03:23.837 ********* 2021-03-09T08:43:03.667 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-03-09T08:43:03.696 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:03 +0000 (0:00:00.039) 0:03:23.877 ********* 2021-03-09T08:43:04.033 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-09T08:43:04.036 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:39:07 UTC", "ActiveEnterTimestampMonotonic": "12909282", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target ntpd.service -.mount system.slice sntp.service systemd-journald.socket ntpdate.service sysinit.target systemd-tmpfiles-setup.service tmp.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:39:06 UTC", "AssertTimestampMonotonic": "11779606", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:39:06 UTC", "ConditionTimestampMonotonic": "11779545", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service ntpd.service shutdown.target", "ControlGro 2021-03-09T08:43:04.038 INFO:teuthology.task.ansible.out:up": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "912", "ExecMainStartTimestamp": "Tue 2021-03-09 08:39:07 UTC", "ExecMainStartTimestampMonotonic": "12843012", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[Tue 2021-03-09 08:39:06 UTC] ; stop_time=[Tue 2021-03-09 08:39:07 UTC] ; pid=904 ; code=exited ; status=0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[Tue 2021-03-09 08:39:07 UTC] ; stop_time=[Tue 2021-03-09 08:39:07 UTC] ; pid=966 ; code=exited ; status=0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:39:06 UTC", "InactiveExitTimestampMonotonic": "11851039", "InvocationID": "459df63085b64add842210bfa799d56d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "Limit 2021-03-09T08:43:04.039 INFO:teuthology.task.ansible.out:AS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127253", "LimitNPROCSoft": "127253", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127253", "LimitSIGPENDINGSoft": "127253", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "912", "MemoryAccounting": "yes", "MemoryCurrent": "3313664", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "R 2021-03-09T08:43:04.041 INFO:teuthology.task.ansible.out:equires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:39:07 UTC", "StateChangeTimestampMonotonic": "12909282", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:39:07 UTC", "WatchdogTimestampMonotonic": "12843030", "WatchdogUSec": "0"}} ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-09 08:39:05 UTC", "ActiveEnterTimestampMonotonic": "12301861", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "tmp.mount sntp.service systemd-journa 2021-03-09T08:43:04.043 INFO:teuthology.task.ansible.out:ld.socket sysinit.target ntpd.service systemd-tmpfiles-setup.service basic.target ntpdate.service system.slice -.mount", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-09 08:39:04 UTC", "AssertTimestampMonotonic": "10860749", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-09 08:39:04 UTC", "ConditionTimestampMonotonic": "10860681", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": 2021-03-09T08:43:04.045 INFO:teuthology.task.ansible.out:"", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "819", "ExecMainStartTimestamp": "Tue 2021-03-09 08:39:04 UTC", "ExecMainStartTimestampMonotonic": "10969658", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[Tue 2021-03-09 08:39:04 UTC] ; stop_time=[Tue 2021-03-09 08:39:04 UTC] ; pid=810 ; code=exited ; status=0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[Tue 2021-03-09 08:39:04 UTC] ; stop_time=[Tue 2021-03-09 08:39:05 UTC] ; pid=823 ; code=exited ; status=0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-09 08:39:04 UTC", "InactiveExitTimestampMonotonic": "10876710", "InvocationID": "8235dc46089548539fa4a81bcecf530a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65 2021-03-09T08:43:04.046 INFO:teuthology.task.ansible.out:536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127252", "LimitNPROCSoft": "127252", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127252", "LimitSIGPENDINGSoft": "127252", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "819", "MemoryAccounting": "yes", "MemoryCurrent": "3309568", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUS 2021-03-09T08:43:04.048 INFO:teuthology.task.ansible.out:ec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-09 08:39:05 UTC", "StateChangeTimestampMonotonic": "12301861", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203604", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-09 08:39:04 UTC", "WatchdogTimestampMonotonic": "10969696", "WatchdogUSec": "0"}} 2021-03-09T08:43:04.066 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.369) 0:03:24.246 ********* 2021-03-09T08:43:04.089 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.022) 0:03:24.269 ********* 2021-03-09T08:43:04.105 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:43:04.107 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T08:43:04.135 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.046) 0:03:24.315 ********* 2021-03-09T08:43:04.143 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-09T08:43:04.170 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.035) 0:03:24.350 ********* 2021-03-09T08:43:04.178 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:43:04.180 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-03-09T08:43:04.205 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.035) 0:03:24.386 ********* 2021-03-09T08:43:04.214 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-03-09T08:43:04.240 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.034) 0:03:24.420 ********* 2021-03-09T08:43:04.248 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:04.273 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.032) 0:03:24.453 ********* 2021-03-09T08:43:04.281 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:04.306 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.032) 0:03:24.486 ********* 2021-03-09T08:43:04.313 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-09T08:43:04.315 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:04.338 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.031) 0:03:24.518 ********* 2021-03-09T08:43:04.345 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:04.370 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.032) 0:03:24.550 ********* 2021-03-09T08:43:04.377 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-09T08:43:04.379 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:04.403 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.033) 0:03:24.583 ********* 2021-03-09T08:43:04.410 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:04.438 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.032) 0:03:24.616 ********* 2021-03-09T08:43:04.601 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-09T08:43:04.603 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:43:04.626 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.189) 0:03:24.806 ********* 2021-03-09T08:43:04.668 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-03-09T08:43:04.698 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:04 +0000 (0:00:00.072) 0:03:24.879 ********* 2021-03-09T08:43:05.110 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-03-09T08:43:05.134 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:05 +0000 (0:00:00.435) 0:03:25.314 ********* 2021-03-09T08:43:05.184 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-03-09T08:43:05.207 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:05 +0000 (0:00:00.073) 0:03:25.388 ********* 2021-03-09T08:43:06.459 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-09T08:43:06.461 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-09T08:43:06.482 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:01.274) 0:03:26.662 ********* 2021-03-09T08:43:06.499 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-09T08:43:06.522 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:00.039) 0:03:26.702 ********* 2021-03-09T08:43:06.531 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-09T08:43:06.532 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:06.553 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:00.031) 0:03:26.734 ********* 2021-03-09T08:43:06.560 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:06.583 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:00.029) 0:03:26.764 ********* 2021-03-09T08:43:06.591 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-09T08:43:06.593 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-09T08:43:06.616 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:00.032) 0:03:26.796 ********* 2021-03-09T08:43:06.622 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-09T08:43:06.647 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:00.031) 0:03:26.828 ********* 2021-03-09T08:43:06.803 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-09T08:43:06.805 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T08:43:06.827 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:00.179) 0:03:27.008 ********* 2021-03-09T08:43:06.835 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-09T08:43:06.861 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:00.033) 0:03:27.041 ********* 2021-03-09T08:43:06.868 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-09T08:43:06.870 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:06.894 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:00.032) 0:03:27.074 ********* 2021-03-09T08:43:06.902 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:06.970 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:06 +0000 (0:00:00.075) 0:03:27.150 ********* 2021-03-09T08:43:06.978 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-09T08:43:06.980 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T08:43:07.005 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.035) 0:03:27.186 ********* 2021-03-09T08:43:07.055 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-09T08:43:07.057 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-09T08:43:07.078 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.072) 0:03:27.258 ********* 2021-03-09T08:43:07.100 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.021) 0:03:27.280 ********* 2021-03-09T08:43:07.117 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-09T08:43:07.119 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.142 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.042) 0:03:27.322 ********* 2021-03-09T08:43:07.372 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-09T08:43:07.374 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-09T08:43:07.377 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-09T08:43:07.401 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.258) 0:03:27.580 ********* 2021-03-09T08:43:07.421 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.020) 0:03:27.601 ********* 2021-03-09T08:43:07.622 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-09T08:43:07.624 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T08:43:07.647 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.226) 0:03:27.827 ********* 2021-03-09T08:43:07.656 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-09T08:43:07.680 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.032) 0:03:27.860 ********* 2021-03-09T08:43:07.688 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-09T08:43:07.689 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.716 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.036) 0:03:27.896 ********* 2021-03-09T08:43:07.727 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.759 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.042) 0:03:27.939 ********* 2021-03-09T08:43:07.767 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-09T08:43:07.768 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.780 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.781 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.808 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.048) 0:03:27.988 ********* 2021-03-09T08:43:07.819 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.820 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.846 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.038) 0:03:28.026 ********* 2021-03-09T08:43:07.853 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-09T08:43:07.855 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi099.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.866 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.867 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.878 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.904 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.058) 0:03:28.084 ********* 2021-03-09T08:43:07.913 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.915 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.916 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi195.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.941 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.037) 0:03:28.122 ********* 2021-03-09T08:43:07.948 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-09T08:43:07.949 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:07.975 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:07 +0000 (0:00:00.033) 0:03:28.155 ********* 2021-03-09T08:43:07.982 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.009 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.034) 0:03:28.189 ********* 2021-03-09T08:43:08.016 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-09T08:43:08.017 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.045 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.036) 0:03:28.225 ********* 2021-03-09T08:43:08.053 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.078 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.032) 0:03:28.258 ********* 2021-03-09T08:43:08.085 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-09T08:43:08.087 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.112 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.034) 0:03:28.292 ********* 2021-03-09T08:43:08.120 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.145 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.033) 0:03:28.325 ********* 2021-03-09T08:43:08.152 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-09T08:43:08.154 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.179 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.033) 0:03:28.359 ********* 2021-03-09T08:43:08.186 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.213 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.034) 0:03:28.393 ********* 2021-03-09T08:43:08.221 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-09T08:43:08.223 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.250 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.036) 0:03:28.430 ********* 2021-03-09T08:43:08.264 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.290 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.039) 0:03:28.470 ********* 2021-03-09T08:43:08.297 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:43:08.299 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.332 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.042) 0:03:28.512 ********* 2021-03-09T08:43:08.340 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.367 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.034) 0:03:28.547 ********* 2021-03-09T08:43:08.375 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-09T08:43:08.376 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.403 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.036) 0:03:28.583 ********* 2021-03-09T08:43:08.410 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.437 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.034) 0:03:28.618 ********* 2021-03-09T08:43:08.444 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-09T08:43:08.446 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.471 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.033) 0:03:28.651 ********* 2021-03-09T08:43:08.478 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:08.505 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:08 +0000 (0:00:00.034) 0:03:28.685 ********* 2021-03-09T08:43:08.694 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-09T08:43:08.696 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005263", "end": "2021-03-09 08:43:08.651653", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 08:43:08.646390", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:08.751 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005094", "end": "2021-03-09 08:43:08.714414", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-09 08:43:08.709320", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:08.866 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002947", "end": "2021-03-09 08:43:08.837895", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 08:43:08.834948", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:09.034 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002743", "end": "2021-03-09 08:43:09.009201", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 08:43:09.006458", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:09.211 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.015089", "end": "2021-03-09 08:43:09.187725", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 08:43:09.172636", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:09.238 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:09 +0000 (0:00:00.732) 0:03:29.418 ********* 2021-03-09T08:43:09.267 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-09T08:43:09.290 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:09 +0000 (0:00:00.051) 0:03:29.470 ********* 2021-03-09T08:43:09.490 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-09T08:43:09.492 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-03-09T08:43:09.937 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-09T08:43:09.939 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:01.004126", "end": "2021-03-09 08:43:09.901716", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-09 08:43:08.897590", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:10.105 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002858", "end": "2021-03-09 08:43:10.080068", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-09 08:43:10.077210", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:11.309 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:01.036439", "end": "2021-03-09 08:43:11.277841", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-09 08:43:10.241402", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:11.343 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:11 +0000 (0:00:02.052) 0:03:31.523 ********* 2021-03-09T08:43:11.375 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-09T08:43:11.398 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:11 +0000 (0:00:00.054) 0:03:31.578 ********* 2021-03-09T08:43:11.593 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-09T08:43:11.595 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-03-09T08:43:11.673 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-09T08:43:11.702 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:11 +0000 (0:00:00.304) 0:03:31.882 ********* 2021-03-09T08:43:11.722 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:11 +0000 (0:00:00.020) 0:03:31.902 ********* 2021-03-09T08:43:12.338 INFO:teuthology.task.ansible.out: 2021-03-09T08:43:12.340 INFO:teuthology.task.ansible.out:TASK [Gathering Facts] ********************************************************* ok: [smithi099.front.sepia.ceph.com] 2021-03-09T08:43:12.393 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:12 +0000 (0:00:00.670) 0:03:32.573 ********* 2021-03-09T08:43:12.432 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:43:12.434 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:12.454 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:12 +0000 (0:00:00.061) 0:03:32.634 ********* 2021-03-09T08:43:12.493 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:43:12.495 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-03-09T08:43:12.516 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:12 +0000 (0:00:00.062) 0:03:32.697 ********* 2021-03-09T08:43:12.556 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** ok: [smithi099.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-03-09T08:43:12.583 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:12 +0000 (0:00:00.066) 0:03:32.763 ********* 2021-03-09T08:43:13.350 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi195.front.sepia.ceph.com] 2021-03-09T08:43:13.398 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:13 +0000 (0:00:00.815) 0:03:33.579 ********* 2021-03-09T08:43:13.427 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:43:13.428 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:13.449 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:13 +0000 (0:00:00.050) 0:03:33.629 ********* 2021-03-09T08:43:13.487 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:43:13.489 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-03-09T08:43:13.513 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:13 +0000 (0:00:00.063) 0:03:33.693 ********* 2021-03-09T08:43:13.561 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** ok: [smithi195.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-03-09T08:43:13.584 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:13 +0000 (0:00:00.071) 0:03:33.764 ********* 2021-03-09T08:43:13.920 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-09T08:43:13.921 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:43:13.947 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:13 +0000 (0:00:00.363) 0:03:34.127 ********* 2021-03-09T08:43:13.976 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:43:13.978 INFO:teuthology.task.ansible.out:ok: [smithi099.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-03-09T08:43:13.999 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:13 +0000 (0:00:00.051) 0:03:34.179 ********* 2021-03-09T08:43:14.027 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:43:14.029 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:14.049 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:14 +0000 (0:00:00.050) 0:03:34.229 ********* 2021-03-09T08:43:14.267 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-09T08:43:14.268 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:43:14.438 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:43:14.469 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:14 +0000 (0:00:00.419) 0:03:34.649 ********* 2021-03-09T08:43:14.922 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** ok: [smithi195.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-09T08:43:14.945 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:14 +0000 (0:00:00.476) 0:03:35.126 ********* 2021-03-09T08:43:14.974 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:43:14.975 INFO:teuthology.task.ansible.out:ok: [smithi195.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-03-09T08:43:14.997 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:14 +0000 (0:00:00.051) 0:03:35.177 ********* 2021-03-09T08:43:15.005 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-09T08:43:15.007 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279394.5395775-22844-148533929107976/source", "state": "file", "uid": 0} 2021-03-09T08:43:15.029 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-09T08:43:15.031 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:15.052 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:15 +0000 (0:00:00.055) 0:03:35.233 ********* 2021-03-09T08:43:15.282 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-09T08:43:15.284 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:43:15.453 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-09T08:43:15.478 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:15 +0000 (0:00:00.425) 0:03:35.658 ********* 2021-03-09T08:43:15.507 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-09T08:43:15.509 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279395.025961-22844-88648080830515/source", "state": "file", "uid": 0} 2021-03-09T08:43:15.538 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:15 +0000 (0:00:00.059) 0:03:35.718 ********* 2021-03-09T08:43:16.016 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279395.550536-23028-164812499864393/source", "state": "file", "uid": 0} 2021-03-09T08:43:16.511 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615279396.059489-23028-168558115563282/source", "state": "file", "uid": 0} 2021-03-09T08:43:16.537 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:16 +0000 (0:00:00.998) 0:03:36.717 ********* 2021-03-09T08:43:18.620 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T08:43:18.634 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-lydojhz_-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-lydojhz_-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-03-09T08:43:18.636 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-03-09T08:43:18.650 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:18 +0000 (0:00:02.112) 0:03:38.830 ********* 2021-03-09T08:43:18.680 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:18.706 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:18 +0000 (0:00:00.056) 0:03:38.886 ********* 2021-03-09T08:43:18.755 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-03-09T08:43:18.757 INFO:teuthology.task.ansible.out:skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:18.784 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:18 +0000 (0:00:00.077) 0:03:38.964 ********* 2021-03-09T08:43:19.148 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-09T08:43:19.151 INFO:teuthology.task.ansible.out:changed: [smithi099.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.165395", "end": "2021-03-09 08:43:19.124298", "rc": 0, "start": "2021-03-09 08:43:18.958903", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:19.171 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:19 +0000 (0:00:00.387) 0:03:39.351 ********* 2021-03-09T08:43:19.210 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi099.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:19.697 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-09T08:43:19.698 INFO:teuthology.task.ansible.out:changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-tfwzelhg-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-tfwzelhg-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-03-09T08:43:19.701 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-03-09T08:43:19.718 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:19 +0000 (0:00:00.547) 0:03:39.898 ********* 2021-03-09T08:43:19.747 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-09T08:43:19.749 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:19.769 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:19 +0000 (0:00:00.051) 0:03:39.950 ********* 2021-03-09T08:43:19.807 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-03-09T08:43:19.809 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:19.830 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:19 +0000 (0:00:00.060) 0:03:40.010 ********* 2021-03-09T08:43:20.172 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** changed: [smithi195.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.156815", "end": "2021-03-09 08:43:20.151878", "rc": 0, "start": "2021-03-09 08:43:19.995063", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-09T08:43:20.194 INFO:teuthology.task.ansible.out:Tuesday 09 March 2021 08:43:20 +0000 (0:00:00.364) 0:03:40.374 ********* 2021-03-09T08:43:20.231 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-09T08:43:20.233 INFO:teuthology.task.ansible.out:skipping: [smithi195.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-09T08:43:20.300 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-09T08:43:20.301 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-03-09T08:43:20.303 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi099.front.sepia.ceph.com : ok=127 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=3 smithi195.front.sepia.ceph.com : ok=127 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=3 Tuesday 09 March 2021 08:43:20 +0000 (0:00:00.109) 0:03:40.483 ********* =============================================================================== 2021-03-09T08:43:20.305 INFO:teuthology.task.ansible.out:users : Update authorized_keys using the keys repo --------------------- 29.03s Create all admin users with sudo access. ------------------------------- 27.92s common : nrpe - Remove SELinux policy package -------------------------- 16.18s common : nrpe - Load SELinux policy package ---------------------------- 15.88s Remove revoked users ---------------------------------------------------- 9.50s testnode : Zap all non-root disks --------------------------------------- 6.97s common : Install Kerberos Packages (RedHat) ----------------------------- 4.39s testnode : Install packages --------------------------------------------- 3.07s testnode : Ensure ceph packages are not present. ------------------------ 2.85s testnode : Rebuild rpmdb ------------------------------------------------ 2.35s testnode : Ensure ceph dependency packages are not present. ------------- 2.17s container-host : Install registries-conf-ctl ---------------------------- 2.11s Gathering Facts --------------------------------------------------------- 1.70s testnode : Create logical volume(s) ------------------------------------- 1.44s users : Update authorized_keys for each user with literal keys ---------- 1.39s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.33s testnode : Install packages via pip ------------------------------------- 1.27s common : Make sure smartmontools is installed --------------------------- 1.24s testnode : Configure local mirrorlists ---------------------------------- 1.24s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.24s 2021-03-09T08:43:20.684 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2021-03-09T08:43:20.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:43:20.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:43:20.760 DEBUG:teuthology.orchestra.run.smithi099:> true 2021-03-09T08:43:20.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-09T08:43:20.857 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-09T08:43:20.926 DEBUG:teuthology.orchestra.run.smithi195:> true 2021-03-09T08:43:21.019 INFO:teuthology.run_tasks:Running task clock... 2021-03-09T08:43:21.046 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-09T08:43:21.046 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T08:43:21.048 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T08:43:21.083 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-03-09T08:43:21.099 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-03-09T08:43:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2021-03-09T08:43:21.149 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2021-03-09T08:43:21.163 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-03-09T08:43:21.177 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-03-09T08:43:21.233 INFO:teuthology.orchestra.run.smithi099.stderr:bash: ntpq: command not found 2021-03-09T08:43:21.236 INFO:teuthology.orchestra.run.smithi099.stdout:210 Number of sources = 4 2021-03-09T08:43:21.236 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-03-09T08:43:21.236 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2021-03-09T08:43:21.236 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-09T08:43:21.237 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-09T08:43:21.237 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-09T08:43:21.237 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-09T08:43:21.237 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-09T08:43:21.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-09T08:43:21.273 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-03-09T08:43:21.288 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-03-09T08:43:21.323 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2021-03-09T08:43:21.334 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2021-03-09T08:43:21.348 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-03-09T08:43:21.362 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-03-09T08:43:21.421 INFO:teuthology.orchestra.run.smithi195.stderr:bash: ntpq: command not found 2021-03-09T08:43:21.425 INFO:teuthology.orchestra.run.smithi195.stdout:210 Number of sources = 4 2021-03-09T08:43:21.425 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-03-09T08:43:21.425 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2021-03-09T08:43:21.426 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-09T08:43:21.426 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-09T08:43:21.426 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-09T08:43:21.426 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-09T08:43:21.427 INFO:teuthology.run_tasks:Running task install... 2021-03-09T08:43:21.447 DEBUG:teuthology.task.install:project ceph 2021-03-09T08:43:21.448 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}} 2021-03-09T08:43:21.448 DEBUG:teuthology.task.install:config {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'} 2021-03-09T08:43:21.449 INFO:teuthology.task.install:Using flavor: basic 2021-03-09T08:43:21.455 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-09T08:43:21.456 INFO:teuthology.task.install:extra packages: [] 2021-03-09T08:43:21.456 DEBUG:teuthology.orchestra.run.smithi099:> sudo lsb_release -is 2021-03-09T08:43:21.489 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS 2021-03-09T08:43:21.490 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-09T08:43:21.492 DEBUG:teuthology.orchestra.run.smithi195:> sudo lsb_release -is 2021-03-09T08:43:21.493 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'tag': None, 'wait_for_package': False} 2021-03-09T08:43:21.494 DEBUG:teuthology.orchestra.run.smithi099:> sudo lsb_release -is 2021-03-09T08:43:21.526 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS 2021-03-09T08:43:21.527 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-09T08:43:21.528 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'tag': None, 'wait_for_package': False} 2021-03-09T08:43:21.528 DEBUG:teuthology.orchestra.run.smithi195:> sudo lsb_release -is 2021-03-09T08:43:21.562 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS 2021-03-09T08:43:21.563 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-09T08:43:21.564 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:43:21.598 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS 2021-03-09T08:43:21.600 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-09T08:43:21.600 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:43:21.712 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/centos/8/flavors/default/ 2021-03-09T08:43:21.713 INFO:teuthology.task.install.rpm:Package version is 17.0.0-1742.g89fb622b 2021-03-09T08:43:21.749 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/centos/8/flavors/default/ 2021-03-09T08:43:21.750 INFO:teuthology.task.install.rpm:Package version is 17.0.0-1742.g89fb622b 2021-03-09T08:43:21.831 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-03-09T08:43:21.832 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:43:21.832 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-03-09T08:43:21.859 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-03-09T08:43:21.860 DEBUG:teuthology.orchestra.run.smithi099:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/89fb622b10ca2e58fe0c913a8956377a886e2ab4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-03-09T08:43:21.872 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-kefu-testing-2021-03-09-1428/89fb622b10ca2e58fe0c913a8956377a886e2ab4/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-03-09T08:43:21.872 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:43:21.873 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-03-09T08:43:21.903 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-03-09T08:43:21.904 DEBUG:teuthology.orchestra.run.smithi195:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/89fb622b10ca2e58fe0c913a8956377a886e2ab4/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-03-09T08:43:21.935 DEBUG:teuthology.orchestra.run.smithi099:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-03-09T08:43:21.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-03-09T08:43:22.020 DEBUG:teuthology.orchestra.run.smithi099:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-03-09T08:43:22.056 DEBUG:teuthology.orchestra.run.smithi195:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-03-09T08:43:22.084 INFO:teuthology.orchestra.run.smithi099.stdout:check_obsoletes = 1 2021-03-09T08:43:22.085 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum clean all 2021-03-09T08:43:22.122 INFO:teuthology.orchestra.run.smithi195.stdout:check_obsoletes = 1 2021-03-09T08:43:22.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum clean all 2021-03-09T08:43:22.444 INFO:teuthology.orchestra.run.smithi099.stdout:63 files removed 2021-03-09T08:43:22.478 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-radosgw 2021-03-09T08:43:22.498 INFO:teuthology.orchestra.run.smithi195.stdout:63 files removed 2021-03-09T08:43:22.533 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-radosgw 2021-03-09T08:43:23.127 INFO:teuthology.orchestra.run.smithi099.stdout:ceph packages for x86_64 304 kB/s | 75 kB 00:00 2021-03-09T08:43:23.181 INFO:teuthology.orchestra.run.smithi195.stdout:ceph packages for x86_64 305 kB/s | 75 kB 00:00 2021-03-09T08:43:23.358 INFO:teuthology.orchestra.run.smithi099.stdout:ceph noarch packages 75 kB/s | 15 kB 00:00 2021-03-09T08:43:23.412 INFO:teuthology.orchestra.run.smithi195.stdout:ceph noarch packages 76 kB/s | 15 kB 00:00 2021-03-09T08:43:23.579 INFO:teuthology.orchestra.run.smithi099.stdout:ceph source packages 7.2 kB/s | 1.4 kB 00:00 2021-03-09T08:43:23.633 INFO:teuthology.orchestra.run.smithi195.stdout:ceph source packages 7.2 kB/s | 1.4 kB 00:00 2021-03-09T08:43:23.802 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - AppStream 32 MB/s | 6.3 MB 00:00 2021-03-09T08:43:23.903 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-8 - AppStream 27 MB/s | 6.3 MB 00:00 2021-03-09T08:43:24.900 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - Base 24 MB/s | 2.3 MB 00:00 2021-03-09T08:43:25.018 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-8 - Base 21 MB/s | 2.3 MB 00:00 2021-03-09T08:43:25.338 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS-8 - Extras 144 kB/s | 9.2 kB 00:00 2021-03-09T08:43:25.472 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-8 - Extras 143 kB/s | 9.2 kB 00:00 2021-03-09T08:43:25.896 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Linux 8 - AppStream 17 MB/s | 6.3 MB 00:00 2021-03-09T08:43:26.012 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Linux 8 - AppStream 10 MB/s | 6.3 MB 00:00 2021-03-09T08:44:26.736 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Linux 8 - BaseOS 39 kB/s | 2.3 MB 00:59 2021-03-09T08:44:27.477 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Linux 8 - Extras 25 kB/s | 9.2 kB 00:00 2021-03-09T08:44:28.054 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 17 MB/s | 8.9 MB 00:00 2021-03-09T08:44:29.923 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-03-09T08:44:31.507 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:44:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:44:31.523 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:44:31.523 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:44:31.524 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:44:31.524 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 9.1 M 2021-03-09T08:44:31.524 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrading: 2021-03-09T08:44:31.524 INFO:teuthology.orchestra.run.smithi099.stdout: librados2 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.3 M 2021-03-09T08:44:31.525 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.4 M 2021-03-09T08:44:31.525 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2021-03-09T08:44:31.525 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 5.1 M 2021-03-09T08:44:31.525 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 19 M 2021-03-09T08:44:31.525 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 23 k 2021-03-09T08:44:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-03-09T08:44:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs x86_64 2.8-1.el8 epel 306 k 2021-03-09T08:44:31.526 INFO:teuthology.orchestra.run.smithi099.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-03-09T08:44:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 748 k 2021-03-09T08:44:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-03-09T08:44:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-03-09T08:44:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 457 k 2021-03-09T08:44:31.527 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-03-09T08:44:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.7 M 2021-03-09T08:44:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-03-09T08:44:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-03-09T08:44:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse 2021-03-09T08:44:31.528 INFO:teuthology.orchestra.run.smithi099.stdout: x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 42 k 2021-03-09T08:44:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 73 k 2021-03-09T08:44:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 176 k 2021-03-09T08:44:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-03-09T08:44:31.529 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 365 k 2021-03-09T08:44:31.530 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 346 k 2021-03-09T08:44:31.530 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 110 k 2021-03-09T08:44:31.530 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:44:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:44:31.530 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:44:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:Install 22 Packages 2021-03-09T08:44:31.531 INFO:teuthology.orchestra.run.smithi099.stdout:Upgrade 2 Packages 2021-03-09T08:44:31.531 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:44:31.532 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 48 M 2021-03-09T08:44:31.532 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:44:32.207 INFO:teuthology.orchestra.run.smithi099.stdout:(1/24): ceph-base-17.0.0-1742.g89fb622b.el8.x86 7.5 MB/s | 5.1 MB 00:00 2021-03-09T08:44:33.392 INFO:teuthology.orchestra.run.smithi099.stdout:(2/24): ceph-selinux-17.0.0-1742.g89fb622b.el8. 517 kB/s | 23 kB 00:00 2021-03-09T08:44:33.392 INFO:teuthology.orchestra.run.smithi099.stdout:(3/24): libcephfs2-17.0.0-1742.g89fb622b.el8.x8 6.1 MB/s | 748 kB 00:00 2021-03-09T08:44:33.392 INFO:teuthology.orchestra.run.smithi099.stdout:(4/24): libradosstriper1-17.0.0-1742.g89fb622b. 6.6 MB/s | 457 kB 00:00 2021-03-09T08:44:33.392 INFO:teuthology.orchestra.run.smithi099.stdout:(5/24): ceph-radosgw-17.0.0-1742.g89fb622b.el8. 8.3 MB/s | 9.1 MB 00:01 2021-03-09T08:44:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:(6/24): python3-ceph-argparse-17.0.0-1742.g89fb 692 kB/s | 42 kB 00:00 2021-03-09T08:44:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:(7/24): python3-ceph-common-17.0.0-1742.g89fb62 1.1 MB/s | 73 kB 00:00 2021-03-09T08:44:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:(8/24): librgw2-17.0.0-1742.g89fb622b.el8.x86_6 11 MB/s | 3.7 MB 00:00 2021-03-09T08:44:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:(9/24): python3-cephfs-17.0.0-1742.g89fb622b.el 2.5 MB/s | 176 kB 00:00 2021-03-09T08:44:33.393 INFO:teuthology.orchestra.run.smithi099.stdout:(10/24): python3-rados-17.0.0-1742.g89fb622b.el 6.5 MB/s | 365 kB 00:00 2021-03-09T08:44:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:(11/24): python3-rbd-17.0.0-1742.g89fb622b.el8. 5.4 MB/s | 346 kB 00:00 2021-03-09T08:44:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:(12/24): python3-rgw-17.0.0-1742.g89fb622b.el8. 2.0 MB/s | 110 kB 00:00 2021-03-09T08:44:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 9.9 MB/s | 353 kB 00:00 2021-03-09T08:44:33.394 INFO:teuthology.orchestra.run.smithi099.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 28 MB/s | 44 kB 00:00 2021-03-09T08:44:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:(15/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 27 MB/s | 47 kB 00:00 2021-03-09T08:44:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:(16/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.9 MB/s | 259 kB 00:00 2021-03-09T08:44:33.395 INFO:teuthology.orchestra.run.smithi099.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 1.4 MB/s | 116 kB 00:00 2021-03-09T08:44:33.396 INFO:teuthology.orchestra.run.smithi099.stdout:(18/24): gperftools-libs-2.8-1.el8.x86_64.rpm 3.1 MB/s | 306 kB 00:00 2021-03-09T08:44:33.396 INFO:teuthology.orchestra.run.smithi099.stdout:(19/24): leveldb-1.22-1.el8.x86_64.rpm 5.5 MB/s | 181 kB 00:00 2021-03-09T08:44:33.396 INFO:teuthology.orchestra.run.smithi099.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 3.1 MB/s | 59 kB 00:00 2021-03-09T08:44:33.396 INFO:teuthology.orchestra.run.smithi099.stdout:(21/24): ceph-common-17.0.0-1742.g89fb622b.el8. 12 MB/s | 19 MB 00:01 2021-03-09T08:44:33.396 INFO:teuthology.orchestra.run.smithi099.stdout:(22/24): libunwind-1.3.1-3.el8.x86_64.rpm 439 kB/s | 75 kB 00:00 2021-03-09T08:44:33.397 INFO:teuthology.orchestra.run.smithi099.stdout:(23/24): librados2-17.0.0-1742.g89fb622b.el8.x8 14 MB/s | 3.3 MB 00:00 2021-03-09T08:44:33.397 INFO:teuthology.orchestra.run.smithi099.stdout:(24/24): librbd1-17.0.0-1742.g89fb622b.el8.x86_ 18 MB/s | 3.4 MB 00:00 2021-03-09T08:44:33.397 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:44:33.397 INFO:teuthology.orchestra.run.smithi099.stdout:Total 26 MB/s | 48 MB 00:01 2021-03-09T08:44:33.398 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:44:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:44:33.417 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:44:34.005 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:44:34.008 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:44:35.086 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:44:35.291 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-03-09T08:44:35.521 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-03-09T08:44:35.706 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-03-09T08:44:36.131 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-03-09T08:44:36.356 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 4/26 2021-03-09T08:44:36.472 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 4/26 2021-03-09T08:44:36.721 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rados-2:17.0.0-1742.g89fb622b.el8.x86_64 5/26 2021-03-09T08:44:36.922 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 6/26 2021-03-09T08:44:37.325 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 6/26 2021-03-09T08:44:37.550 INFO:teuthology.orchestra.run.smithi099.stdout: Upgrading : librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 7/26 2021-03-09T08:44:37.607 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 7/26 2021-03-09T08:44:37.813 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/26 2021-03-09T08:44:38.006 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-03-09T08:44:38.400 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-03-09T08:44:38.555 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librgw2-2:17.0.0-1742.g89fb622b.el8.x86_64 10/26 2021-03-09T08:44:38.631 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librgw2-2:17.0.0-1742.g89fb622b.el8.x86_64 10/26 2021-03-09T08:44:38.786 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-03-09T08:44:38.906 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-argparse-2:17.0.0-1742.g89fb622b.el8. 12/26 2021-03-09T08:44:39.092 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cephfs-2:17.0.0-1742.g89fb622b.el8.x86_64 13/26 2021-03-09T08:44:39.289 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rgw-2:17.0.0-1742.g89fb622b.el8.x86_64 14/26 2021-03-09T08:44:39.473 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rbd-2:17.0.0-1742.g89fb622b.el8.x86_64 15/26 2021-03-09T08:44:39.672 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libradosstriper1-2:17.0.0-1742.g89fb622b.el8.x86_6 16/26 2021-03-09T08:44:39.743 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libradosstriper1-2:17.0.0-1742.g89fb622b.el8.x86_6 16/26 2021-03-09T08:44:39.956 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-03-09T08:44:40.130 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : gperftools-libs-2.8-1.el8.x86_64 18/26 2021-03-09T08:44:40.339 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-03-09T08:44:40.539 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-ceph-common-2:17.0.0-1742.g89fb622b.el8.x8 20/26 2021-03-09T08:44:44.473 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 21/26 2021-03-09T08:44:44.680 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 21/26 2021-03-09T08:44:45.367 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 21/26 2021-03-09T08:44:45.680 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 22/26 2021-03-09T08:44:45.778 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 22/26 2021-03-09T08:44:45.937 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-selinux-2:17.0.0-1742.g89fb622b.el8.x86_64 23/26 2021-03-09T08:45:02.782 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Linux 8 - BaseOS 24 kB/s | 2.3 MB 01:35 2021-03-09T08:45:02.861 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-selinux-2:17.0.0-1742.g89fb622b.el8.x86_64 23/26 2021-03-09T08:45:02.862 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /sys 2021-03-09T08:45:02.862 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /proc 2021-03-09T08:45:02.862 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2021-03-09T08:45:02.862 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /run 2021-03-09T08:45:02.863 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /mnt 2021-03-09T08:45:02.863 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /var/tmp 2021-03-09T08:45:02.863 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /home 2021-03-09T08:45:02.863 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /tmp 2021-03-09T08:45:02.863 INFO:teuthology.orchestra.run.smithi099.stdout:skipping the directory /dev 2021-03-09T08:45:02.864 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:04.034 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-radosgw-2:17.0.0-1742.g89fb622b.el8.x86_64 24/26 2021-03-09T08:45:04.118 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-1742.g89fb622b.el8.x86_64 24/26 2021-03-09T08:45:04.118 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-03-09T08:45:04.311 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-03-09T08:45:04.311 INFO:teuthology.orchestra.run.smithi099.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-03-09T08:45:05.043 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-03-09T08:45:05.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 1/26 2021-03-09T08:45:05.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 2/26 2021-03-09T08:45:05.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-radosgw-2:17.0.0-1742.g89fb622b.el8.x86_64 3/26 2021-03-09T08:45:05.044 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-selinux-2:17.0.0-1742.g89fb622b.el8.x86_64 4/26 2021-03-09T08:45:05.045 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 5/26 2021-03-09T08:45:05.045 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libradosstriper1-2:17.0.0-1742.g89fb622b.el8.x86_6 6/26 2021-03-09T08:45:05.045 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librgw2-2:17.0.0-1742.g89fb622b.el8.x86_64 7/26 2021-03-09T08:45:05.045 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-argparse-2:17.0.0-1742.g89fb622b.el8. 8/26 2021-03-09T08:45:05.046 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-ceph-common-2:17.0.0-1742.g89fb622b.el8.x8 9/26 2021-03-09T08:45:05.046 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cephfs-2:17.0.0-1742.g89fb622b.el8.x86_64 10/26 2021-03-09T08:45:05.046 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rados-2:17.0.0-1742.g89fb622b.el8.x86_64 11/26 2021-03-09T08:45:05.046 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rbd-2:17.0.0-1742.g89fb622b.el8.x86_64 12/26 2021-03-09T08:45:05.047 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rgw-2:17.0.0-1742.g89fb622b.el8.x86_64 13/26 2021-03-09T08:45:05.047 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-03-09T08:45:05.047 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-03-09T08:45:05.047 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-03-09T08:45:05.048 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 17/26 2021-03-09T08:45:05.048 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-03-09T08:45:05.048 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : gperftools-libs-2.8-1.el8.x86_64 19/26 2021-03-09T08:45:05.048 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-03-09T08:45:05.049 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-03-09T08:45:05.049 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-03-09T08:45:05.049 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 23/26 2021-03-09T08:45:05.049 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-03-09T08:45:05.050 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 25/26 2021-03-09T08:45:05.423 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Linux 8 - Extras 4.1 kB/s | 9.2 kB 00:02 2021-03-09T08:45:05.735 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-03-09T08:45:05.735 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:45:05.938 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 19 MB/s | 8.9 MB 00:00 2021-03-09T08:45:06.003 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:06.003 INFO:teuthology.orchestra.run.smithi099.stdout:Upgraded: 2021-03-09T08:45:06.003 INFO:teuthology.orchestra.run.smithi099.stdout: librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.004 INFO:teuthology.orchestra.run.smithi099.stdout: librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.004 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:06.004 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:45:06.004 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.005 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.005 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-radosgw-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.005 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-selinux-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.005 INFO:teuthology.orchestra.run.smithi099.stdout: fmt-6.2.1-1.el8.x86_64 2021-03-09T08:45:06.005 INFO:teuthology.orchestra.run.smithi099.stdout: gperftools-libs-2.8-1.el8.x86_64 2021-03-09T08:45:06.006 INFO:teuthology.orchestra.run.smithi099.stdout: leveldb-1.22-1.el8.x86_64 2021-03-09T08:45:06.006 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.006 INFO:teuthology.orchestra.run.smithi099.stdout: liboath-2.6.2-3.el8.x86_64 2021-03-09T08:45:06.006 INFO:teuthology.orchestra.run.smithi099.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-03-09T08:45:06.007 INFO:teuthology.orchestra.run.smithi099.stdout: libradosstriper1-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.007 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-03-09T08:45:06.007 INFO:teuthology.orchestra.run.smithi099.stdout: librgw2-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.007 INFO:teuthology.orchestra.run.smithi099.stdout: libunwind-1.3.1-3.el8.x86_64 2021-03-09T08:45:06.007 INFO:teuthology.orchestra.run.smithi099.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-03-09T08:45:06.008 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.008 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.009 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cephfs-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.009 INFO:teuthology.orchestra.run.smithi099.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-03-09T08:45:06.009 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rados-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.009 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rbd-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.010 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rgw-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:06.010 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:06.010 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:45:06.154 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-test 2021-03-09T08:45:06.741 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:37 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:45:07.437 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:45:07.438 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:07.438 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:45:07.438 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:07.439 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:45:07.439 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 40 M 2021-03-09T08:45:07.439 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2021-03-09T08:45:07.439 INFO:teuthology.orchestra.run.smithi099.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-03-09T08:45:07.439 INFO:teuthology.orchestra.run.smithi099.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-03-09T08:45:07.440 INFO:teuthology.orchestra.run.smithi099.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-03-09T08:45:07.440 INFO:teuthology.orchestra.run.smithi099.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-03-09T08:45:07.440 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-03-09T08:45:07.440 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:07.441 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:45:07.441 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:07.441 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2021-03-09T08:45:07.441 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:07.442 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 41 M 2021-03-09T08:45:07.442 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 229 M 2021-03-09T08:45:07.442 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:45:07.488 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.9 MB/s | 187 kB 00:00 2021-03-09T08:45:07.490 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 3.2 MB/s | 161 kB 00:00 2021-03-09T08:45:07.493 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): socat-1.7.3.3-2.el8.x86_64.rpm 56 MB/s | 302 kB 00:00 2021-03-09T08:45:07.496 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): libxslt-1.1.32-5.el8.x86_64.rpm 37 MB/s | 250 kB 00:00 2021-03-09T08:45:07.558 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 1.0 MB/s | 69 kB 00:00 2021-03-09T08:45:07.821 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-03-09T08:45:09.393 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:45:09.410 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:45:09.411 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:45:09.411 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:45:09.411 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:45:09.411 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 9.1 M 2021-03-09T08:45:09.412 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrading: 2021-03-09T08:45:09.412 INFO:teuthology.orchestra.run.smithi195.stdout: librados2 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.3 M 2021-03-09T08:45:09.412 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.4 M 2021-03-09T08:45:09.412 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2021-03-09T08:45:09.413 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 5.1 M 2021-03-09T08:45:09.413 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 19 M 2021-03-09T08:45:09.413 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 23 k 2021-03-09T08:45:09.414 INFO:teuthology.orchestra.run.smithi195.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-03-09T08:45:09.414 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs x86_64 2.8-1.el8 epel 306 k 2021-03-09T08:45:09.414 INFO:teuthology.orchestra.run.smithi195.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-03-09T08:45:09.414 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 748 k 2021-03-09T08:45:09.414 INFO:teuthology.orchestra.run.smithi195.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-03-09T08:45:09.415 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-03-09T08:45:09.416 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 457 k 2021-03-09T08:45:09.416 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-03-09T08:45:09.416 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2 x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.7 M 2021-03-09T08:45:09.416 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-03-09T08:45:09.417 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-03-09T08:45:09.417 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse 2021-03-09T08:45:09.417 INFO:teuthology.orchestra.run.smithi195.stdout: x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 42 k 2021-03-09T08:45:09.417 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 73 k 2021-03-09T08:45:09.417 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 176 k 2021-03-09T08:45:09.418 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-03-09T08:45:09.418 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 365 k 2021-03-09T08:45:09.418 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 346 k 2021-03-09T08:45:09.418 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 110 k 2021-03-09T08:45:09.419 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:09.419 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:45:09.419 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:45:09.419 INFO:teuthology.orchestra.run.smithi195.stdout:Install 22 Packages 2021-03-09T08:45:09.419 INFO:teuthology.orchestra.run.smithi195.stdout:Upgrade 2 Packages 2021-03-09T08:45:09.420 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:09.420 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 48 M 2021-03-09T08:45:09.421 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:45:10.003 INFO:teuthology.orchestra.run.smithi195.stdout:(1/24): ceph-base-17.0.0-1742.g89fb622b.el8.x86 8.7 MB/s | 5.1 MB 00:00 2021-03-09T08:45:10.512 INFO:teuthology.orchestra.run.smithi195.stdout:(2/24): ceph-selinux-17.0.0-1742.g89fb622b.el8. 475 kB/s | 23 kB 00:00 2021-03-09T08:45:10.513 INFO:teuthology.orchestra.run.smithi195.stdout:(3/24): libcephfs2-17.0.0-1742.g89fb622b.el8.x8 9.3 MB/s | 748 kB 00:00 2021-03-09T08:45:10.513 INFO:teuthology.orchestra.run.smithi195.stdout:(4/24): libradosstriper1-17.0.0-1742.g89fb622b. 6.5 MB/s | 457 kB 00:00 2021-03-09T08:45:10.577 INFO:teuthology.orchestra.run.smithi195.stdout:(5/24): librgw2-17.0.0-1742.g89fb622b.el8.x86_6 9.7 MB/s | 3.7 MB 00:00 2021-03-09T08:45:10.621 INFO:teuthology.orchestra.run.smithi195.stdout:(6/24): python3-ceph-argparse-17.0.0-1742.g89fb 956 kB/s | 42 kB 00:00 2021-03-09T08:45:10.666 INFO:teuthology.orchestra.run.smithi195.stdout:(7/24): python3-ceph-common-17.0.0-1742.g89fb62 1.6 MB/s | 73 kB 00:00 2021-03-09T08:45:10.714 INFO:teuthology.orchestra.run.smithi195.stdout:(8/24): python3-cephfs-17.0.0-1742.g89fb622b.el 3.6 MB/s | 176 kB 00:00 2021-03-09T08:45:10.775 INFO:teuthology.orchestra.run.smithi195.stdout:(9/24): ceph-radosgw-17.0.0-1742.g89fb622b.el8. 6.7 MB/s | 9.1 MB 00:01 2021-03-09T08:45:10.777 INFO:teuthology.orchestra.run.smithi195.stdout:(10/24): python3-rados-17.0.0-1742.g89fb622b.el 5.6 MB/s | 365 kB 00:00 2021-03-09T08:45:10.813 INFO:teuthology.orchestra.run.smithi195.stdout:(11/24): python3-rbd-17.0.0-1742.g89fb622b.el8. 8.9 MB/s | 346 kB 00:00 2021-03-09T08:45:10.821 INFO:teuthology.orchestra.run.smithi195.stdout:(12/24): python3-rgw-17.0.0-1742.g89fb622b.el8. 2.5 MB/s | 110 kB 00:00 2021-03-09T08:45:10.853 INFO:teuthology.orchestra.run.smithi195.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 8.8 MB/s | 353 kB 00:00 2021-03-09T08:45:10.854 INFO:teuthology.orchestra.run.smithi195.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 24 MB/s | 44 kB 00:00 2021-03-09T08:45:10.856 INFO:teuthology.orchestra.run.smithi195.stdout:(15/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.3 MB/s | 259 kB 00:00 2021-03-09T08:45:10.857 INFO:teuthology.orchestra.run.smithi195.stdout:(16/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 24 MB/s | 47 kB 00:00 2021-03-09T08:45:10.937 INFO:teuthology.orchestra.run.smithi195.stdout:(17/24): fmt-6.2.1-1.el8.x86_64.rpm 1.4 MB/s | 116 kB 00:00 2021-03-09T08:45:10.953 INFO:teuthology.orchestra.run.smithi195.stdout:(18/24): gperftools-libs-2.8-1.el8.x86_64.rpm 3.1 MB/s | 306 kB 00:00 2021-03-09T08:45:10.970 INFO:teuthology.orchestra.run.smithi195.stdout:(19/24): leveldb-1.22-1.el8.x86_64.rpm 5.3 MB/s | 181 kB 00:00 2021-03-09T08:45:10.971 INFO:teuthology.orchestra.run.smithi195.stdout:(20/24): liboath-2.6.2-3.el8.x86_64.rpm 3.2 MB/s | 59 kB 00:00 2021-03-09T08:45:10.988 INFO:teuthology.orchestra.run.smithi195.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 4.4 MB/s | 75 kB 00:00 2021-03-09T08:45:11.289 INFO:teuthology.orchestra.run.smithi195.stdout:(22/24): librados2-17.0.0-1742.g89fb622b.el8.x8 10 MB/s | 3.3 MB 00:00 2021-03-09T08:45:11.460 INFO:teuthology.orchestra.run.smithi195.stdout:(23/24): ceph-common-17.0.0-1742.g89fb622b.el8. 9.5 MB/s | 19 MB 00:02 2021-03-09T08:45:11.520 INFO:teuthology.orchestra.run.smithi195.stdout:(24/24): librbd1-17.0.0-1742.g89fb622b.el8.x86_ 6.4 MB/s | 3.4 MB 00:00 2021-03-09T08:45:11.523 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:45:11.524 INFO:teuthology.orchestra.run.smithi195.stdout:Total 23 MB/s | 48 MB 00:02 2021-03-09T08:45:11.528 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:45:11.586 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:45:11.586 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:45:12.227 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:45:12.230 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:45:13.371 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:45:13.519 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-03-09T08:45:13.745 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-03-09T08:45:13.829 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): ceph-test-17.0.0-1742.g89fb622b.el8.x86_ 6.3 MB/s | 40 MB 00:06 2021-03-09T08:45:13.833 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:45:13.833 INFO:teuthology.orchestra.run.smithi099.stdout:Total 6.5 MB/s | 41 MB 00:06 2021-03-09T08:45:13.838 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:45:13.861 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:45:13.862 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:45:13.938 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-03-09T08:45:14.115 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:45:14.118 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:45:14.326 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-03-09T08:45:14.529 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 4/26 2021-03-09T08:45:14.624 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 4/26 2021-03-09T08:45:14.683 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:45:14.886 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rados-2:17.0.0-1742.g89fb622b.el8.x86_64 5/26 2021-03-09T08:45:15.080 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 6/26 2021-03-09T08:45:15.482 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 6/26 2021-03-09T08:45:15.708 INFO:teuthology.orchestra.run.smithi195.stdout: Upgrading : librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 7/26 2021-03-09T08:45:15.774 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 7/26 2021-03-09T08:45:15.952 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/6 2021-03-09T08:45:15.970 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/26 2021-03-09T08:45:16.138 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-03-09T08:45:16.412 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2021-03-09T08:45:16.552 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-03-09T08:45:16.602 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/6 2021-03-09T08:45:16.704 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librgw2-2:17.0.0-1742.g89fb622b.el8.x86_64 10/26 2021-03-09T08:45:16.775 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librgw2-2:17.0.0-1742.g89fb622b.el8.x86_64 10/26 2021-03-09T08:45:16.780 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-03-09T08:45:16.873 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/6 2021-03-09T08:45:16.943 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-03-09T08:45:17.105 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-argparse-2:17.0.0-1742.g89fb622b.el8. 12/26 2021-03-09T08:45:17.274 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cephfs-2:17.0.0-1742.g89fb622b.el8.x86_64 13/26 2021-03-09T08:45:17.496 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rgw-2:17.0.0-1742.g89fb622b.el8.x86_64 14/26 2021-03-09T08:45:17.722 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rbd-2:17.0.0-1742.g89fb622b.el8.x86_64 15/26 2021-03-09T08:45:17.896 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libradosstriper1-2:17.0.0-1742.g89fb622b.el8.x86_6 16/26 2021-03-09T08:45:18.015 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libradosstriper1-2:17.0.0-1742.g89fb622b.el8.x86_6 16/26 2021-03-09T08:45:18.222 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-03-09T08:45:18.404 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : gperftools-libs-2.8-1.el8.x86_64 18/26 2021-03-09T08:45:18.613 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-03-09T08:45:18.771 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-ceph-common-2:17.0.0-1742.g89fb622b.el8.x8 20/26 2021-03-09T08:45:21.250 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2021-03-09T08:45:21.460 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-test-2:17.0.0-1742.g89fb622b.el8.x86_64 6/6 2021-03-09T08:45:21.695 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-test-2:17.0.0-1742.g89fb622b.el8.x86_64 6/6 2021-03-09T08:45:21.696 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-test-2:17.0.0-1742.g89fb622b.el8.x86_64 1/6 2021-03-09T08:45:21.696 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2021-03-09T08:45:21.696 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-03-09T08:45:21.696 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 4/6 2021-03-09T08:45:21.697 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 5/6 2021-03-09T08:45:22.028 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-03-09T08:45:22.028 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:45:22.275 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:22.276 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:45:22.276 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-test-2:17.0.0-1742.g89fb622b.el8.x86_64 jq-1.5-12.el8.x86_64 2021-03-09T08:45:22.276 INFO:teuthology.orchestra.run.smithi099.stdout: libxslt-1.1.32-5.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-03-09T08:45:22.277 INFO:teuthology.orchestra.run.smithi099.stdout: socat-1.7.3.3-2.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-03-09T08:45:22.277 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:22.277 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:45:22.421 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph 2021-03-09T08:45:22.785 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 21/26 2021-03-09T08:45:22.935 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:00:53 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:45:23.070 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 21/26 2021-03-09T08:45:23.606 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:45:23.609 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================= 2021-03-09T08:45:23.609 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:45:23.609 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================= 2021-03-09T08:45:23.609 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:45:23.610 INFO:teuthology.orchestra.run.smithi099.stdout: ceph x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 6.4 k 2021-03-09T08:45:23.610 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2021-03-09T08:45:23.610 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 2.2 M 2021-03-09T08:45:23.610 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 1.4 M 2021-03-09T08:45:23.611 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 198 k 2021-03-09T08:45:23.611 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.5 M 2021-03-09T08:45:23.611 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 16 M 2021-03-09T08:45:23.611 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asn1crypto noarch 0.24.0-3.el8 CentOS-Base 181 k 2021-03-09T08:45:23.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-03-09T08:45:23.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-03-09T08:45:23.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-03-09T08:45:23.612 INFO:teuthology.orchestra.run.smithi099.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-03-09T08:45:23.613 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-03-09T08:45:23.613 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-03-09T08:45:23.613 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography x86_64 2.3-3.el8 CentOS-Base 510 k 2021-03-09T08:45:23.613 INFO:teuthology.orchestra.run.smithi099.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-03-09T08:45:23.613 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-03-09T08:45:23.614 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-03-09T08:45:23.614 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-03-09T08:45:23.614 INFO:teuthology.orchestra.run.smithi099.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-03-09T08:45:23.614 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-03-09T08:45:23.615 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-03-09T08:45:23.615 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-03-09T08:45:23.615 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-03-09T08:45:23.615 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-03-09T08:45:23.616 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 CentOS-AppStream 103 k 2021-03-09T08:45:23.617 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-03-09T08:45:23.617 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-03-09T08:45:23.617 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-03-09T08:45:23.617 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-03-09T08:45:23.618 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-03-09T08:45:23.618 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-03-09T08:45:23.618 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-03-09T08:45:23.619 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-03-09T08:45:23.619 INFO:teuthology.orchestra.run.smithi099.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-03-09T08:45:23.619 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-03-09T08:45:23.620 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-03-09T08:45:23.620 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-03-09T08:45:23.620 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-03-09T08:45:23.620 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-03-09T08:45:23.622 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-03-09T08:45:23.622 INFO:teuthology.orchestra.run.smithi099.stdout:Installing weak dependencies: 2021-03-09T08:45:23.622 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-03-09T08:45:23.622 INFO:teuthology.orchestra.run.smithi099.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-03-09T08:45:23.623 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:23.623 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:45:23.623 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================= 2021-03-09T08:45:23.624 INFO:teuthology.orchestra.run.smithi099.stdout:Install 42 Packages 2021-03-09T08:45:23.624 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:23.625 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 29 M 2021-03-09T08:45:23.625 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 120 M 2021-03-09T08:45:23.625 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:45:23.714 INFO:teuthology.orchestra.run.smithi099.stdout:(1/42): ceph-17.0.0-1742.g89fb622b.el8.x86_64.r 65 kB/s | 6.4 kB 00:00 2021-03-09T08:45:23.760 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 21/26 2021-03-09T08:45:23.962 INFO:teuthology.orchestra.run.smithi099.stdout:(2/42): ceph-mgr-17.0.0-1742.g89fb622b.el8.x86_ 4.2 MB/s | 1.4 MB 00:00 2021-03-09T08:45:23.989 INFO:teuthology.orchestra.run.smithi099.stdout:(3/42): ceph-mds-17.0.0-1742.g89fb622b.el8.x86_ 5.9 MB/s | 2.2 MB 00:00 2021-03-09T08:45:24.046 INFO:teuthology.orchestra.run.smithi099.stdout:(4/42): ceph-mgr-modules-core-17.0.0-1742.g89fb 3.5 MB/s | 198 kB 00:00 2021-03-09T08:45:24.047 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 22/26 2021-03-09T08:45:24.077 INFO:teuthology.orchestra.run.smithi099.stdout:(5/42): python3-html5lib-0.999999999-6.el8.noar 6.5 MB/s | 214 kB 00:00 2021-03-09T08:45:24.092 INFO:teuthology.orchestra.run.smithi099.stdout:(6/42): ceph-mon-17.0.0-1742.g89fb622b.el8.x86_ 9.2 MB/s | 3.5 MB 00:00 2021-03-09T08:45:24.109 INFO:teuthology.orchestra.run.smithi099.stdout:(7/42): python3-lxml-4.2.3-1.el8.x86_64.rpm 47 MB/s | 1.5 MB 00:00 2021-03-09T08:45:24.111 INFO:teuthology.orchestra.run.smithi099.stdout:(8/42): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2021-03-09T08:45:24.113 INFO:teuthology.orchestra.run.smithi099.stdout:(9/42): python3-pyOpenSSL-18.0.0-1.el8.noarch.r 54 MB/s | 103 kB 00:00 2021-03-09T08:45:24.117 INFO:teuthology.orchestra.run.smithi099.stdout:(10/42): python3-pytz-2017.2-9.el8.noarch.rpm 39 MB/s | 54 kB 00:00 2021-03-09T08:45:24.118 INFO:teuthology.orchestra.run.smithi099.stdout:(11/42): python3-webencodings-0.5.1-6.el8.noarc 24 MB/s | 27 kB 00:00 2021-03-09T08:45:24.122 INFO:teuthology.orchestra.run.smithi099.stdout:(12/42): python3-werkzeug-0.12.2-4.el8.noarch.r 72 MB/s | 457 kB 00:00 2021-03-09T08:45:24.125 INFO:teuthology.orchestra.run.smithi099.stdout:(13/42): python3-asn1crypto-0.24.0-3.el8.noarch 51 MB/s | 181 kB 00:00 2021-03-09T08:45:24.129 INFO:teuthology.orchestra.run.smithi099.stdout:(14/42): python3-mako-1.0.6-13.el8.noarch.rpm 4.2 MB/s | 157 kB 00:00 2021-03-09T08:45:24.130 INFO:teuthology.orchestra.run.smithi099.stdout:(15/42): python3-cffi-1.11.5-5.el8.x86_64.rpm 53 MB/s | 237 kB 00:00 2021-03-09T08:45:24.135 INFO:teuthology.orchestra.run.smithi099.stdout:(16/42): python3-chardet-3.0.4-7.el8.noarch.rpm 31 MB/s | 195 kB 00:00 2021-03-09T08:45:24.138 INFO:teuthology.orchestra.run.smithi099.stdout:(17/42): python3-cryptography-2.3-3.el8.x86_64. 64 MB/s | 510 kB 00:00 2021-03-09T08:45:24.139 INFO:teuthology.orchestra.run.smithi099.stdout:(18/42): python3-idna-2.5-5.el8.noarch.rpm 26 MB/s | 97 kB 00:00 2021-03-09T08:45:24.140 INFO:teuthology.orchestra.run.smithi099.stdout:(19/42): python3-pycparser-2.14-14.el8.noarch.r 50 MB/s | 109 kB 00:00 2021-03-09T08:45:24.141 INFO:teuthology.orchestra.run.smithi099.stdout:(20/42): python3-pysocks-1.6.8-3.el8.noarch.rpm 20 MB/s | 34 kB 00:00 2021-03-09T08:45:24.143 INFO:teuthology.orchestra.run.smithi099.stdout:(21/42): python3-requests-2.20.0-2.1.el8_1.noar 53 MB/s | 123 kB 00:00 2021-03-09T08:45:24.145 INFO:teuthology.orchestra.run.smithi099.stdout:(22/42): python3-urllib3-1.24.2-4.el8.noarch.rp 41 MB/s | 176 kB 00:00 2021-03-09T08:45:24.175 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 22/26 2021-03-09T08:45:24.222 INFO:teuthology.orchestra.run.smithi099.stdout:(23/42): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 557 kB/s | 44 kB 00:00 2021-03-09T08:45:24.242 INFO:teuthology.orchestra.run.smithi099.stdout:(24/42): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.9 MB/s | 185 kB 00:00 2021-03-09T08:45:24.280 INFO:teuthology.orchestra.run.smithi099.stdout:(25/42): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.9 MB/s | 173 kB 00:00 2021-03-09T08:45:24.282 INFO:teuthology.orchestra.run.smithi099.stdout:(26/42): python3-cherrypy-18.4.0-1.el8.noarch.r 9.4 MB/s | 384 kB 00:00 2021-03-09T08:45:24.300 INFO:teuthology.orchestra.run.smithi099.stdout:(27/42): python3-cssselect-0.9.2-10.el8.noarch. 2.0 MB/s | 40 kB 00:00 2021-03-09T08:45:24.302 INFO:teuthology.orchestra.run.smithi099.stdout:(28/42): python3-jaraco-6.2-6.el8.noarch.rpm 572 kB/s | 11 kB 00:00 2021-03-09T08:45:24.319 INFO:teuthology.orchestra.run.smithi099.stdout:(29/42): python3-jaraco-functools-2.0-4.el8.noa 908 kB/s | 18 kB 00:00 2021-03-09T08:45:24.322 INFO:teuthology.orchestra.run.smithi099.stdout:(30/42): python3-logutils-0.3.5-11.el8.noarch.r 2.4 MB/s | 49 kB 00:00 2021-03-09T08:45:24.328 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-selinux-2:17.0.0-1742.g89fb622b.el8.x86_64 23/26 2021-03-09T08:45:24.339 INFO:teuthology.orchestra.run.smithi099.stdout:(31/42): python3-more-itertools-7.2.0-3.el8.noa 2.9 MB/s | 59 kB 00:00 2021-03-09T08:45:24.359 INFO:teuthology.orchestra.run.smithi099.stdout:(32/42): python3-portend-2.6-1.el8.noarch.rpm 792 kB/s | 16 kB 00:00 2021-03-09T08:45:24.362 INFO:teuthology.orchestra.run.smithi099.stdout:(33/42): python3-pecan-1.3.2-9.el8.noarch.rpm 7.1 MB/s | 283 kB 00:00 2021-03-09T08:45:24.380 INFO:teuthology.orchestra.run.smithi099.stdout:(34/42): python3-simplegeneric-0.8.1-17.el8.noa 956 kB/s | 19 kB 00:00 2021-03-09T08:45:24.381 INFO:teuthology.orchestra.run.smithi099.stdout:(35/42): python3-singledispatch-3.4.0.3-18.el8. 1.2 MB/s | 24 kB 00:00 2021-03-09T08:45:24.398 INFO:teuthology.orchestra.run.smithi099.stdout:(36/42): python3-tempora-1.14.1-5.el8.noarch.rp 1.5 MB/s | 29 kB 00:00 2021-03-09T08:45:24.401 INFO:teuthology.orchestra.run.smithi099.stdout:(37/42): python3-trustme-0.6.0-4.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-03-09T08:45:24.437 INFO:teuthology.orchestra.run.smithi099.stdout:(38/42): python3-waitress-1.2.1-2.el8.1.noarch. 6.0 MB/s | 241 kB 00:00 2021-03-09T08:45:24.440 INFO:teuthology.orchestra.run.smithi099.stdout:(39/42): python3-webob-1.8.5-1.el8.1.noarch.rpm 6.3 MB/s | 251 kB 00:00 2021-03-09T08:45:24.457 INFO:teuthology.orchestra.run.smithi099.stdout:(40/42): python3-webtest-2.0.33-1.el8.noarch.rp 4.2 MB/s | 85 kB 00:00 2021-03-09T08:45:24.459 INFO:teuthology.orchestra.run.smithi099.stdout:(41/42): python3-zc-lockfile-2.0-2.el8.noarch.r 1.1 MB/s | 23 kB 00:00 2021-03-09T08:45:24.671 INFO:teuthology.orchestra.run.smithi099.stdout:(42/42): ceph-osd-17.0.0-1742.g89fb622b.el8.x86 22 MB/s | 16 MB 00:00 2021-03-09T08:45:24.675 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:45:24.676 INFO:teuthology.orchestra.run.smithi099.stdout:Total 27 MB/s | 29 MB 00:01 2021-03-09T08:45:24.689 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:45:24.731 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:45:24.731 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:45:25.071 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:45:25.075 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:45:26.535 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:45:26.723 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/42 2021-03-09T08:45:26.936 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-idna-2.5-5.el8.noarch 2/42 2021-03-09T08:45:27.120 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 3/42 2021-03-09T08:45:27.377 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/42 2021-03-09T08:45:27.560 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/42 2021-03-09T08:45:27.699 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/42 2021-03-09T08:45:27.897 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/42 2021-03-09T08:45:28.090 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/42 2021-03-09T08:45:28.226 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/42 2021-03-09T08:45:28.406 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/42 2021-03-09T08:45:28.577 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/42 2021-03-09T08:45:28.811 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/42 2021-03-09T08:45:29.038 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 13/42 2021-03-09T08:45:29.298 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/42 2021-03-09T08:45:29.497 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/42 2021-03-09T08:45:29.746 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 16/42 2021-03-09T08:45:29.978 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-03-09T08:45:30.224 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 18/42 2021-03-09T08:45:30.571 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 19/42 2021-03-09T08:45:30.823 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 20/42 2021-03-09T08:45:31.085 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 21/42 2021-03-09T08:45:31.294 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 22/42 2021-03-09T08:45:31.572 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/42 2021-03-09T08:45:31.780 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/42 2021-03-09T08:45:32.509 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 25/42 2021-03-09T08:45:32.724 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 26/42 2021-03-09T08:45:33.070 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 27/42 2021-03-09T08:45:33.287 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/42 2021-03-09T08:45:33.483 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 29/42 2021-03-09T08:45:33.680 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 30/42 2021-03-09T08:45:33.866 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 31/42 2021-03-09T08:45:34.130 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-portend-2.6-1.el8.noarch 32/42 2021-03-09T08:45:34.389 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 33/42 2021-03-09T08:45:34.590 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 34/42 2021-03-09T08:45:34.948 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 35/42 2021-03-09T08:45:35.297 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 36/42 2021-03-09T08:45:35.677 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-1742.g89fb622b.el8. 37/42 2021-03-09T08:45:35.852 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 38/42 2021-03-09T08:45:37.266 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 38/42 2021-03-09T08:45:37.621 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-osd-2:17.0.0-1742.g89fb622b.el8.x86_64 39/42 2021-03-09T08:45:37.993 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-osd-2:17.0.0-1742.g89fb622b.el8.x86_64 39/42 2021-03-09T08:45:38.193 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mon-2:17.0.0-1742.g89fb622b.el8.x86_64 40/42 2021-03-09T08:45:38.411 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mon-2:17.0.0-1742.g89fb622b.el8.x86_64 40/42 2021-03-09T08:45:38.592 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mds-2:17.0.0-1742.g89fb622b.el8.x86_64 41/42 2021-03-09T08:45:38.613 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mds-2:17.0.0-1742.g89fb622b.el8.x86_64 41/42 2021-03-09T08:45:38.766 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-2:17.0.0-1742.g89fb622b.el8.x86_64 42/42 2021-03-09T08:45:39.135 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-2:17.0.0-1742.g89fb622b.el8.x86_64 42/42 2021-03-09T08:45:39.136 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-2:17.0.0-1742.g89fb622b.el8.x86_64 1/42 2021-03-09T08:45:39.136 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mds-2:17.0.0-1742.g89fb622b.el8.x86_64 2/42 2021-03-09T08:45:39.136 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 3/42 2021-03-09T08:45:39.137 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mon-2:17.0.0-1742.g89fb622b.el8.x86_64 4/42 2021-03-09T08:45:39.137 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-osd-2:17.0.0-1742.g89fb622b.el8.x86_64 5/42 2021-03-09T08:45:39.138 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-1742.g89fb622b.el8. 6/42 2021-03-09T08:45:39.138 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/42 2021-03-09T08:45:39.138 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/42 2021-03-09T08:45:39.138 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/42 2021-03-09T08:45:39.139 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/42 2021-03-09T08:45:39.139 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 11/42 2021-03-09T08:45:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/42 2021-03-09T08:45:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/42 2021-03-09T08:45:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/42 2021-03-09T08:45:39.140 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 15/42 2021-03-09T08:45:39.141 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 16/42 2021-03-09T08:45:39.141 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-03-09T08:45:39.141 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 18/42 2021-03-09T08:45:39.142 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-idna-2.5-5.el8.noarch 19/42 2021-03-09T08:45:39.142 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 20/42 2021-03-09T08:45:39.142 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 21/42 2021-03-09T08:45:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 22/42 2021-03-09T08:45:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 23/42 2021-03-09T08:45:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 24/42 2021-03-09T08:45:39.143 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 25/42 2021-03-09T08:45:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 26/42 2021-03-09T08:45:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 27/42 2021-03-09T08:45:39.144 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 28/42 2021-03-09T08:45:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 29/42 2021-03-09T08:45:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 30/42 2021-03-09T08:45:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 31/42 2021-03-09T08:45:39.145 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 32/42 2021-03-09T08:45:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 33/42 2021-03-09T08:45:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-portend-2.6-1.el8.noarch 34/42 2021-03-09T08:45:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 35/42 2021-03-09T08:45:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 36/42 2021-03-09T08:45:39.146 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 37/42 2021-03-09T08:45:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 38/42 2021-03-09T08:45:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 39/42 2021-03-09T08:45:39.147 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 40/42 2021-03-09T08:45:39.148 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 41/42 2021-03-09T08:45:39.936 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 42/42 2021-03-09T08:45:39.936 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:45:40.189 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:40.189 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:45:40.190 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:40.190 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mds-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:40.190 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:40.191 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:45:40.191 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mon-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:40.191 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-osd-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:40.191 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-03-09T08:45:40.192 INFO:teuthology.orchestra.run.smithi099.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-03-09T08:45:40.192 INFO:teuthology.orchestra.run.smithi099.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-03-09T08:45:40.192 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-03-09T08:45:40.192 INFO:teuthology.orchestra.run.smithi099.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-03-09T08:45:40.193 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-03-09T08:45:40.193 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-03-09T08:45:40.193 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-03-09T08:45:40.193 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-03-09T08:45:40.194 INFO:teuthology.orchestra.run.smithi099.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-03-09T08:45:40.194 INFO:teuthology.orchestra.run.smithi099.stdout: python3-idna-2.5-5.el8.noarch 2021-03-09T08:45:40.194 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-6.2-6.el8.noarch 2021-03-09T08:45:40.194 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-03-09T08:45:40.195 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-03-09T08:45:40.195 INFO:teuthology.orchestra.run.smithi099.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-03-09T08:45:40.195 INFO:teuthology.orchestra.run.smithi099.stdout: python3-mako-1.0.6-13.el8.noarch 2021-03-09T08:45:40.195 INFO:teuthology.orchestra.run.smithi099.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-03-09T08:45:40.196 INFO:teuthology.orchestra.run.smithi099.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-03-09T08:45:40.196 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-03-09T08:45:40.196 INFO:teuthology.orchestra.run.smithi099.stdout: python3-portend-2.6-1.el8.noarch 2021-03-09T08:45:40.197 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-03-09T08:45:40.197 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pycparser-2.14-14.el8.noarch 2021-03-09T08:45:40.197 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-03-09T08:45:40.197 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pytz-2017.2-9.el8.noarch 2021-03-09T08:45:40.198 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-03-09T08:45:40.198 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-03-09T08:45:40.198 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-03-09T08:45:40.198 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-03-09T08:45:40.199 INFO:teuthology.orchestra.run.smithi099.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-03-09T08:45:40.199 INFO:teuthology.orchestra.run.smithi099.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-03-09T08:45:40.199 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-03-09T08:45:40.200 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-03-09T08:45:40.200 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-03-09T08:45:40.200 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-03-09T08:45:40.200 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-03-09T08:45:40.200 INFO:teuthology.orchestra.run.smithi099.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-03-09T08:45:40.201 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:40.201 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:45:40.333 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-base 2021-03-09T08:45:40.861 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:11 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:45:41.358 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:45:41.422 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-selinux-2:17.0.0-1742.g89fb622b.el8.x86_64 23/26 2021-03-09T08:45:41.422 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /sys 2021-03-09T08:45:41.422 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /proc 2021-03-09T08:45:41.422 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2021-03-09T08:45:41.423 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /run 2021-03-09T08:45:41.423 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /mnt 2021-03-09T08:45:41.423 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /var/tmp 2021-03-09T08:45:41.423 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /home 2021-03-09T08:45:41.423 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /tmp 2021-03-09T08:45:41.424 INFO:teuthology.orchestra.run.smithi195.stdout:skipping the directory /dev 2021-03-09T08:45:41.424 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:41.523 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2021-03-09T08:45:41.524 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:45:41.582 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install cephadm 2021-03-09T08:45:42.106 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:13 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:45:42.568 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-radosgw-2:17.0.0-1742.g89fb622b.el8.x86_64 24/26 2021-03-09T08:45:42.628 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-1742.g89fb622b.el8.x86_64 24/26 2021-03-09T08:45:42.629 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-03-09T08:45:42.814 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:45:42.815 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:42.815 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:45:42.815 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:42.816 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:45:42.816 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 65 k 2021-03-09T08:45:42.816 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:42.816 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:45:42.816 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:42.817 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2021-03-09T08:45:42.817 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:42.817 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 65 k 2021-03-09T08:45:42.817 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 284 k 2021-03-09T08:45:42.818 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:45:42.836 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-03-09T08:45:42.836 INFO:teuthology.orchestra.run.smithi195.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-03-09T08:45:42.963 INFO:teuthology.orchestra.run.smithi099.stdout:cephadm-17.0.0-1742.g89fb622b.el8.noarch.rpm 438 kB/s | 65 kB 00:00 2021-03-09T08:45:42.964 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:45:42.965 INFO:teuthology.orchestra.run.smithi099.stdout:Total 434 kB/s | 65 kB 00:00 2021-03-09T08:45:42.965 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:45:42.972 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:45:42.972 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:45:42.974 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:45:42.977 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:45:43.025 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:45:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-03-09T08:45:43.923 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 1/26 2021-03-09T08:45:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 2/26 2021-03-09T08:45:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-radosgw-2:17.0.0-1742.g89fb622b.el8.x86_64 3/26 2021-03-09T08:45:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-selinux-2:17.0.0-1742.g89fb622b.el8.x86_64 4/26 2021-03-09T08:45:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 5/26 2021-03-09T08:45:43.924 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libradosstriper1-2:17.0.0-1742.g89fb622b.el8.x86_6 6/26 2021-03-09T08:45:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librgw2-2:17.0.0-1742.g89fb622b.el8.x86_64 7/26 2021-03-09T08:45:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-argparse-2:17.0.0-1742.g89fb622b.el8. 8/26 2021-03-09T08:45:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-ceph-common-2:17.0.0-1742.g89fb622b.el8.x8 9/26 2021-03-09T08:45:43.925 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cephfs-2:17.0.0-1742.g89fb622b.el8.x86_64 10/26 2021-03-09T08:45:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rados-2:17.0.0-1742.g89fb622b.el8.x86_64 11/26 2021-03-09T08:45:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rbd-2:17.0.0-1742.g89fb622b.el8.x86_64 12/26 2021-03-09T08:45:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rgw-2:17.0.0-1742.g89fb622b.el8.x86_64 13/26 2021-03-09T08:45:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-03-09T08:45:43.926 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-03-09T08:45:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-03-09T08:45:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 17/26 2021-03-09T08:45:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-03-09T08:45:43.927 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : gperftools-libs-2.8-1.el8.x86_64 19/26 2021-03-09T08:45:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-03-09T08:45:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-03-09T08:45:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-03-09T08:45:43.928 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 23/26 2021-03-09T08:45:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-03-09T08:45:43.929 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 25/26 2021-03-09T08:45:44.551 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-03-09T08:45:44.551 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:45:44.849 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:44.849 INFO:teuthology.orchestra.run.smithi195.stdout:Upgraded: 2021-03-09T08:45:44.849 INFO:teuthology.orchestra.run.smithi195.stdout: librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.849 INFO:teuthology.orchestra.run.smithi195.stdout: librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.850 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:44.850 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:45:44.850 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.850 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.851 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-radosgw-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.851 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-selinux-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.851 INFO:teuthology.orchestra.run.smithi195.stdout: fmt-6.2.1-1.el8.x86_64 2021-03-09T08:45:44.851 INFO:teuthology.orchestra.run.smithi195.stdout: gperftools-libs-2.8-1.el8.x86_64 2021-03-09T08:45:44.852 INFO:teuthology.orchestra.run.smithi195.stdout: leveldb-1.22-1.el8.x86_64 2021-03-09T08:45:44.852 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.852 INFO:teuthology.orchestra.run.smithi195.stdout: liboath-2.6.2-3.el8.x86_64 2021-03-09T08:45:44.852 INFO:teuthology.orchestra.run.smithi195.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-03-09T08:45:44.852 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-03-09T08:45:44.854 INFO:teuthology.orchestra.run.smithi195.stdout: librgw2-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: libunwind-1.3.1-3.el8.x86_64 2021-03-09T08:45:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-03-09T08:45:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-argparse-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cephfs-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-03-09T08:45:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rados-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rbd-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.857 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rgw-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:44.857 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:44.858 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:45:44.964 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-test 2021-03-09T08:45:45.025 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/1 2021-03-09T08:45:45.271 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/1 2021-03-09T08:45:45.448 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/1 2021-03-09T08:45:45.558 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:38 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:45:45.666 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/1 2021-03-09T08:45:45.666 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:45:45.908 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:45.909 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:45:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:45:45.909 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:45.910 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:45:46.074 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-immutable-object-cache 2021-03-09T08:45:46.278 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:45:46.279 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:45:46.279 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:45:46.280 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:45:46.280 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:45:46.280 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 40 M 2021-03-09T08:45:46.280 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2021-03-09T08:45:46.281 INFO:teuthology.orchestra.run.smithi195.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-03-09T08:45:46.281 INFO:teuthology.orchestra.run.smithi195.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-03-09T08:45:46.281 INFO:teuthology.orchestra.run.smithi195.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-03-09T08:45:46.281 INFO:teuthology.orchestra.run.smithi195.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-03-09T08:45:46.281 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-03-09T08:45:46.282 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:46.282 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:45:46.282 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:45:46.282 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2021-03-09T08:45:46.283 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:46.283 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 41 M 2021-03-09T08:45:46.284 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 229 M 2021-03-09T08:45:46.284 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:45:46.316 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.7 MB/s | 161 kB 00:00 2021-03-09T08:45:46.321 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): socat-1.7.3.3-2.el8.x86_64.rpm 58 MB/s | 302 kB 00:00 2021-03-09T08:45:46.325 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.3 MB/s | 187 kB 00:00 2021-03-09T08:45:46.326 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): libxslt-1.1.32-5.el8.x86_64.rpm 47 MB/s | 250 kB 00:00 2021-03-09T08:45:46.437 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 619 kB/s | 69 kB 00:00 2021-03-09T08:45:46.619 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:17 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:45:47.278 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:45:47.279 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:47.280 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2021-03-09T08:45:47.280 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:47.280 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:45:47.280 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 155 k 2021-03-09T08:45:47.281 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:47.281 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:45:47.281 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:47.281 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2021-03-09T08:45:47.281 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:47.282 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 155 k 2021-03-09T08:45:47.282 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 587 k 2021-03-09T08:45:47.282 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:45:47.764 INFO:teuthology.orchestra.run.smithi099.stdout:ceph-immutable-object-cache-17.0.0-1742.g89fb62 321 kB/s | 155 kB 00:00 2021-03-09T08:45:47.765 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:45:47.766 INFO:teuthology.orchestra.run.smithi099.stdout:Total 320 kB/s | 155 kB 00:00 2021-03-09T08:45:47.766 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:45:47.779 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:45:47.780 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:45:47.829 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:45:47.832 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:45:47.940 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:45:48.123 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-1742.g89fb622b. 1/1 2021-03-09T08:45:48.219 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): ceph-test-17.0.0-1742.g89fb622b.el8.x86_ 21 MB/s | 40 MB 00:01 2021-03-09T08:45:48.224 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:45:48.224 INFO:teuthology.orchestra.run.smithi195.stdout:Total 21 MB/s | 41 MB 00:01 2021-03-09T08:45:48.233 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:45:48.253 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:45:48.253 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:45:48.485 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-1742.g89fb622b. 1/1 2021-03-09T08:45:48.504 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:45:48.507 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:45:49.043 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-1742.g89fb622b. 1/1 2021-03-09T08:45:49.043 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:45:49.179 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:45:49.560 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/6 2021-03-09T08:45:49.568 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:49.568 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:45:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-immutable-object-cache-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:45:49.569 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:49.569 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:45:49.642 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr 2021-03-09T08:45:49.737 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2021-03-09T08:45:49.936 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/6 2021-03-09T08:45:50.133 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-03-09T08:45:50.180 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:21 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:45:50.221 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/6 2021-03-09T08:45:50.714 INFO:teuthology.orchestra.run.smithi099.stdout:Package ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:45:50.876 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:45:50.877 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2021-03-09T08:45:50.878 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:45:50.931 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-dashboard 2021-03-09T08:45:51.455 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:22 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:45:52.134 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:45:52.134 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:52.135 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:45:52.135 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:52.135 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:45:52.136 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 2.3 M 2021-03-09T08:45:52.136 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2021-03-09T08:45:52.136 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 23 k 2021-03-09T08:45:52.136 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 8.9 k 2021-03-09T08:45:52.136 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-03-09T08:45:52.137 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-03-09T08:45:52.137 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-03-09T08:45:52.137 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:52.137 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:45:52.138 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:45:52.138 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2021-03-09T08:45:52.138 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:52.139 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 2.5 M 2021-03-09T08:45:52.139 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 15 M 2021-03-09T08:45:52.139 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:45:52.245 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): ceph-prometheus-alerts-17.0.0-1742.g89fb 82 kB/s | 8.9 kB 00:00 2021-03-09T08:45:52.259 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): ceph-grafana-dashboards-17.0.0-1742.g89f 187 kB/s | 23 kB 00:00 2021-03-09T08:45:52.276 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2021-03-09T08:45:52.318 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 568 kB/s | 34 kB 00:00 2021-03-09T08:45:52.372 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.0 MB/s | 196 kB 00:00 2021-03-09T08:45:52.427 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): ceph-mgr-dashboard-17.0.0-1742.g89fb622b 7.8 MB/s | 2.3 MB 00:00 2021-03-09T08:45:52.430 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:45:52.430 INFO:teuthology.orchestra.run.smithi099.stdout:Total 8.7 MB/s | 2.5 MB 00:00 2021-03-09T08:45:52.431 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:45:52.440 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:45:52.441 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:45:52.624 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:45:52.627 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:45:52.876 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:45:53.114 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-03-09T08:45:53.324 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-03-09T08:45:53.524 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-03-09T08:45:53.671 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-1742.g89fb622b.el8.n 4/6 2021-03-09T08:45:54.686 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-1742.g89fb622b.el8. 5/6 2021-03-09T08:45:54.857 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2021-03-09T08:45:55.688 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-1742.g89fb622b.el8.noarc 6/6 2021-03-09T08:45:56.102 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-1742.g89fb622b.el8.noarc 6/6 2021-03-09T08:45:56.102 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-1742.g89fb622b.el8. 1/6 2021-03-09T08:45:56.102 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-1742.g89fb622b.el8.noarc 2/6 2021-03-09T08:45:56.103 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-1742.g89fb622b.el8.n 3/6 2021-03-09T08:45:56.103 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-03-09T08:45:56.103 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-03-09T08:45:56.260 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-test-2:17.0.0-1742.g89fb622b.el8.x86_64 6/6 2021-03-09T08:45:56.364 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-03-09T08:45:56.364 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:45:56.541 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-test-2:17.0.0-1742.g89fb622b.el8.x86_64 6/6 2021-03-09T08:45:56.541 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-test-2:17.0.0-1742.g89fb622b.el8.x86_64 1/6 2021-03-09T08:45:56.542 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2021-03-09T08:45:56.542 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-03-09T08:45:56.542 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 4/6 2021-03-09T08:45:56.542 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 5/6 2021-03-09T08:45:56.581 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:56.582 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:45:56.582 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-grafana-dashboards-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:45:56.582 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:45:56.582 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-prometheus-alerts-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:45:56.582 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-03-09T08:45:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-03-09T08:45:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes-2.4.1-12.el8.noarch 2021-03-09T08:45:56.583 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:56.583 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:45:56.652 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-diskprediction-local 2021-03-09T08:45:56.908 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-03-09T08:45:56.908 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:45:57.159 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:57.159 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:45:57.159 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-test-2:17.0.0-1742.g89fb622b.el8.x86_64 jq-1.5-12.el8.x86_64 2021-03-09T08:45:57.159 INFO:teuthology.orchestra.run.smithi195.stdout: libxslt-1.1.32-5.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-03-09T08:45:57.160 INFO:teuthology.orchestra.run.smithi195.stdout: socat-1.7.3.3-2.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-03-09T08:45:57.160 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:57.160 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:45:57.179 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:28 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:45:57.403 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph 2021-03-09T08:45:57.845 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:45:57.846 INFO:teuthology.orchestra.run.smithi099.stdout:===================================================================================================== 2021-03-09T08:45:57.846 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:45:57.846 INFO:teuthology.orchestra.run.smithi099.stdout:===================================================================================================== 2021-03-09T08:45:57.846 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:45:57.847 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 7.4 M 2021-03-09T08:45:57.847 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2021-03-09T08:45:57.847 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-03-09T08:45:57.847 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-03-09T08:45:57.848 INFO:teuthology.orchestra.run.smithi099.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-03-09T08:45:57.848 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-03-09T08:45:57.848 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-03-09T08:45:57.848 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-03-09T08:45:57.849 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-03-09T08:45:57.849 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:57.849 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:45:57.849 INFO:teuthology.orchestra.run.smithi099.stdout:===================================================================================================== 2021-03-09T08:45:57.849 INFO:teuthology.orchestra.run.smithi099.stdout:Install 8 Packages 2021-03-09T08:45:57.850 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:45:57.850 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 35 M 2021-03-09T08:45:57.851 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 194 M 2021-03-09T08:45:57.851 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:45:57.916 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:50 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:45:57.961 INFO:teuthology.orchestra.run.smithi099.stdout:(1/8): openblas-0.3.3-5.el8.x86_64.rpm 38 MB/s | 4.3 MB 00:00 2021-03-09T08:45:58.034 INFO:teuthology.orchestra.run.smithi099.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 52 MB/s | 3.7 MB 00:00 2021-03-09T08:45:58.040 INFO:teuthology.orchestra.run.smithi099.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 39 MB/s | 225 kB 00:00 2021-03-09T08:45:58.051 INFO:teuthology.orchestra.run.smithi099.stdout:(4/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 21 MB/s | 4.3 MB 00:00 2021-03-09T08:45:58.074 INFO:teuthology.orchestra.run.smithi099.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 28 MB/s | 639 kB 00:00 2021-03-09T08:45:58.086 INFO:teuthology.orchestra.run.smithi099.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 14 MB/s | 169 kB 00:00 2021-03-09T08:45:58.323 INFO:teuthology.orchestra.run.smithi099.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 51 MB/s | 15 MB 00:00 2021-03-09T08:45:58.347 INFO:teuthology.orchestra.run.smithi099.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-174 15 MB/s | 7.4 MB 00:00 2021-03-09T08:45:58.350 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:45:58.350 INFO:teuthology.orchestra.run.smithi099.stdout:Total 70 MB/s | 35 MB 00:00 2021-03-09T08:45:58.358 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:45:58.378 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:45:58.378 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:45:58.589 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:45:58.593 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2021-03-09T08:45:58.593 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:45:58.593 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2021-03-09T08:45:58.593 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:45:58.594 INFO:teuthology.orchestra.run.smithi195.stdout: ceph x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 6.4 k 2021-03-09T08:45:58.594 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2021-03-09T08:45:58.594 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 2.2 M 2021-03-09T08:45:58.594 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 1.4 M 2021-03-09T08:45:58.595 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 198 k 2021-03-09T08:45:58.595 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.5 M 2021-03-09T08:45:58.595 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 16 M 2021-03-09T08:45:58.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asn1crypto noarch 0.24.0-3.el8 CentOS-Base 181 k 2021-03-09T08:45:58.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-03-09T08:45:58.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-03-09T08:45:58.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-03-09T08:45:58.596 INFO:teuthology.orchestra.run.smithi195.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-03-09T08:45:58.597 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-03-09T08:45:58.597 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-03-09T08:45:58.597 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography x86_64 2.3-3.el8 CentOS-Base 510 k 2021-03-09T08:45:58.597 INFO:teuthology.orchestra.run.smithi195.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-03-09T08:45:58.598 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-03-09T08:45:58.598 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-03-09T08:45:58.598 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-03-09T08:45:58.598 INFO:teuthology.orchestra.run.smithi195.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-03-09T08:45:58.599 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-03-09T08:45:58.599 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-03-09T08:45:58.599 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-03-09T08:45:58.599 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-03-09T08:45:58.599 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-03-09T08:45:58.600 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL noarch 18.0.0-1.el8 CentOS-AppStream 103 k 2021-03-09T08:45:58.600 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-03-09T08:45:58.600 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-03-09T08:45:58.600 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-03-09T08:45:58.600 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-03-09T08:45:58.601 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-03-09T08:45:58.601 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-03-09T08:45:58.601 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-03-09T08:45:58.601 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-03-09T08:45:58.602 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-03-09T08:45:58.602 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-03-09T08:45:58.602 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-03-09T08:45:58.603 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-03-09T08:45:58.603 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-03-09T08:45:58.603 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-03-09T08:45:58.603 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-03-09T08:45:58.604 INFO:teuthology.orchestra.run.smithi195.stdout:Installing weak dependencies: 2021-03-09T08:45:58.604 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-03-09T08:45:58.604 INFO:teuthology.orchestra.run.smithi195.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-03-09T08:45:58.604 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:58.604 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:45:58.605 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================= 2021-03-09T08:45:58.605 INFO:teuthology.orchestra.run.smithi195.stdout:Install 42 Packages 2021-03-09T08:45:58.605 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:45:58.606 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 29 M 2021-03-09T08:45:58.606 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 120 M 2021-03-09T08:45:58.606 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:45:58.692 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:45:58.695 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:45:58.698 INFO:teuthology.orchestra.run.smithi195.stdout:(1/42): ceph-17.0.0-1742.g89fb622b.el8.x86_64.r 65 kB/s | 6.4 kB 00:00 2021-03-09T08:45:58.905 INFO:teuthology.orchestra.run.smithi195.stdout:(2/42): ceph-mgr-17.0.0-1742.g89fb622b.el8.x86_ 4.7 MB/s | 1.4 MB 00:00 2021-03-09T08:45:58.955 INFO:teuthology.orchestra.run.smithi195.stdout:(3/42): ceph-mds-17.0.0-1742.g89fb622b.el8.x86_ 6.2 MB/s | 2.2 MB 00:00 2021-03-09T08:45:59.018 INFO:teuthology.orchestra.run.smithi195.stdout:(4/42): ceph-mgr-modules-core-17.0.0-1742.g89fb 3.1 MB/s | 198 kB 00:00 2021-03-09T08:45:59.054 INFO:teuthology.orchestra.run.smithi195.stdout:(5/42): python3-html5lib-0.999999999-6.el8.noar 5.8 MB/s | 214 kB 00:00 2021-03-09T08:45:59.080 INFO:teuthology.orchestra.run.smithi195.stdout:(6/42): python3-lxml-4.2.3-1.el8.x86_64.rpm 59 MB/s | 1.5 MB 00:00 2021-03-09T08:45:59.083 INFO:teuthology.orchestra.run.smithi195.stdout:(7/42): python3-mako-1.0.6-13.el8.noarch.rpm 55 MB/s | 157 kB 00:00 2021-03-09T08:45:59.084 INFO:teuthology.orchestra.run.smithi195.stdout:(8/42): python3-markupsafe-0.23-19.el8.x86_64.r 34 MB/s | 39 kB 00:00 2021-03-09T08:45:59.086 INFO:teuthology.orchestra.run.smithi195.stdout:(9/42): python3-pyOpenSSL-18.0.0-1.el8.noarch.r 49 MB/s | 103 kB 00:00 2021-03-09T08:45:59.088 INFO:teuthology.orchestra.run.smithi195.stdout:(10/42): python3-pytz-2017.2-9.el8.noarch.rpm 34 MB/s | 54 kB 00:00 2021-03-09T08:45:59.089 INFO:teuthology.orchestra.run.smithi195.stdout:(11/42): python3-webencodings-0.5.1-6.el8.noarc 23 MB/s | 27 kB 00:00 2021-03-09T08:45:59.096 INFO:teuthology.orchestra.run.smithi195.stdout:(12/42): python3-werkzeug-0.12.2-4.el8.noarch.r 69 MB/s | 457 kB 00:00 2021-03-09T08:45:59.099 INFO:teuthology.orchestra.run.smithi195.stdout:(13/42): python3-asn1crypto-0.24.0-3.el8.noarch 53 MB/s | 181 kB 00:00 2021-03-09T08:45:59.111 INFO:teuthology.orchestra.run.smithi195.stdout:(14/42): ceph-mon-17.0.0-1742.g89fb622b.el8.x86 8.5 MB/s | 3.5 MB 00:00 2021-03-09T08:45:59.112 INFO:teuthology.orchestra.run.smithi195.stdout:(15/42): python3-cffi-1.11.5-5.el8.x86_64.rpm 19 MB/s | 237 kB 00:00 2021-03-09T08:45:59.124 INFO:teuthology.orchestra.run.smithi195.stdout:(16/42): python3-cryptography-2.3-3.el8.x86_64. 76 MB/s | 510 kB 00:00 2021-03-09T08:45:59.126 INFO:teuthology.orchestra.run.smithi195.stdout:(17/42): python3-idna-2.5-5.el8.noarch.rpm 44 MB/s | 97 kB 00:00 2021-03-09T08:45:59.128 INFO:teuthology.orchestra.run.smithi195.stdout:(18/42): python3-pycparser-2.14-14.el8.noarch.r 54 MB/s | 109 kB 00:00 2021-03-09T08:45:59.129 INFO:teuthology.orchestra.run.smithi195.stdout:(19/42): python3-pysocks-1.6.8-3.el8.noarch.rpm 25 MB/s | 34 kB 00:00 2021-03-09T08:45:59.132 INFO:teuthology.orchestra.run.smithi195.stdout:(20/42): python3-requests-2.20.0-2.1.el8_1.noar 56 MB/s | 123 kB 00:00 2021-03-09T08:45:59.135 INFO:teuthology.orchestra.run.smithi195.stdout:(21/42): python3-urllib3-1.24.2-4.el8.noarch.rp 58 MB/s | 176 kB 00:00 2021-03-09T08:45:59.142 INFO:teuthology.orchestra.run.smithi195.stdout:(22/42): python3-chardet-3.0.4-7.el8.noarch.rpm 6.0 MB/s | 195 kB 00:00 2021-03-09T08:45:59.213 INFO:teuthology.orchestra.run.smithi195.stdout:(23/42): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 558 kB/s | 44 kB 00:00 2021-03-09T08:45:59.222 INFO:teuthology.orchestra.run.smithi195.stdout:(24/42): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.3 MB/s | 185 kB 00:00 2021-03-09T08:45:59.256 INFO:teuthology.orchestra.run.smithi195.stdout:(25/42): python3-cherrypy-18.4.0-1.el8.noarch.r 11 MB/s | 384 kB 00:00 2021-03-09T08:45:59.265 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:45:59.271 INFO:teuthology.orchestra.run.smithi195.stdout:(26/42): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.9 MB/s | 173 kB 00:00 2021-03-09T08:45:59.272 INFO:teuthology.orchestra.run.smithi195.stdout:(27/42): python3-cssselect-0.9.2-10.el8.noarch. 2.4 MB/s | 40 kB 00:00 2021-03-09T08:45:59.289 INFO:teuthology.orchestra.run.smithi195.stdout:(28/42): python3-jaraco-functools-2.0-4.el8.noa 1.1 MB/s | 18 kB 00:00 2021-03-09T08:45:59.291 INFO:teuthology.orchestra.run.smithi195.stdout:(29/42): python3-jaraco-6.2-6.el8.noarch.rpm 574 kB/s | 11 kB 00:00 2021-03-09T08:45:59.305 INFO:teuthology.orchestra.run.smithi195.stdout:(30/42): python3-logutils-0.3.5-11.el8.noarch.r 2.9 MB/s | 49 kB 00:00 2021-03-09T08:45:59.311 INFO:teuthology.orchestra.run.smithi195.stdout:(31/42): python3-more-itertools-7.2.0-3.el8.noa 2.9 MB/s | 59 kB 00:00 2021-03-09T08:45:59.331 INFO:teuthology.orchestra.run.smithi195.stdout:(32/42): python3-portend-2.6-1.el8.noarch.rpm 803 kB/s | 16 kB 00:00 2021-03-09T08:45:59.338 INFO:teuthology.orchestra.run.smithi195.stdout:(33/42): python3-pecan-1.3.2-9.el8.noarch.rpm 8.5 MB/s | 283 kB 00:00 2021-03-09T08:45:59.350 INFO:teuthology.orchestra.run.smithi195.stdout:(34/42): python3-simplegeneric-0.8.1-17.el8.noa 960 kB/s | 19 kB 00:00 2021-03-09T08:45:59.355 INFO:teuthology.orchestra.run.smithi195.stdout:(35/42): python3-singledispatch-3.4.0.3-18.el8. 1.4 MB/s | 24 kB 00:00 2021-03-09T08:45:59.370 INFO:teuthology.orchestra.run.smithi195.stdout:(36/42): python3-tempora-1.14.1-5.el8.noarch.rp 1.5 MB/s | 29 kB 00:00 2021-03-09T08:45:59.371 INFO:teuthology.orchestra.run.smithi195.stdout:(37/42): python3-trustme-0.6.0-4.el8.noarch.rpm 1.6 MB/s | 27 kB 00:00 2021-03-09T08:45:59.389 INFO:teuthology.orchestra.run.smithi195.stdout:(38/42): python3-webob-1.8.5-1.el8.1.noarch.rpm 14 MB/s | 251 kB 00:00 2021-03-09T08:45:59.406 INFO:teuthology.orchestra.run.smithi195.stdout:(39/42): python3-webtest-2.0.33-1.el8.noarch.rp 5.0 MB/s | 85 kB 00:00 2021-03-09T08:45:59.409 INFO:teuthology.orchestra.run.smithi195.stdout:(40/42): python3-waitress-1.2.1-2.el8.1.noarch. 6.1 MB/s | 241 kB 00:00 2021-03-09T08:45:59.412 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-03-09T08:45:59.422 INFO:teuthology.orchestra.run.smithi195.stdout:(41/42): python3-zc-lockfile-2.0-2.el8.noarch.r 1.4 MB/s | 23 kB 00:00 2021-03-09T08:45:59.585 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-03-09T08:45:59.658 INFO:teuthology.orchestra.run.smithi195.stdout:(42/42): ceph-osd-17.0.0-1742.g89fb622b.el8.x86 21 MB/s | 16 MB 00:00 2021-03-09T08:45:59.662 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:45:59.663 INFO:teuthology.orchestra.run.smithi195.stdout:Total 27 MB/s | 29 MB 00:01 2021-03-09T08:45:59.676 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:45:59.717 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:45:59.718 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:45:59.762 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-03-09T08:46:00.084 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:46:00.087 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:46:00.252 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-03-09T08:46:00.429 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-03-09T08:46:00.914 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-03-09T08:46:01.287 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-03-09T08:46:01.653 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:46:01.825 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/42 2021-03-09T08:46:02.045 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-03-09T08:46:02.073 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-idna-2.5-5.el8.noarch 2/42 2021-03-09T08:46:02.280 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 3/42 2021-03-09T08:46:02.419 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-03-09T08:46:02.506 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/42 2021-03-09T08:46:02.695 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-03-09T08:46:02.712 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/42 2021-03-09T08:46:02.918 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/42 2021-03-09T08:46:03.125 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/42 2021-03-09T08:46:03.309 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/42 2021-03-09T08:46:03.479 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/42 2021-03-09T08:46:03.634 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/42 2021-03-09T08:46:03.798 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/42 2021-03-09T08:46:04.032 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/42 2021-03-09T08:46:04.423 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 13/42 2021-03-09T08:46:04.537 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/42 2021-03-09T08:46:04.671 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-03-09T08:46:04.741 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/42 2021-03-09T08:46:04.999 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 16/42 2021-03-09T08:46:05.241 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-03-09T08:46:05.456 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 18/42 2021-03-09T08:46:05.799 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-asn1crypto-0.24.0-3.el8.noarch 19/42 2021-03-09T08:46:05.893 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-03-09T08:46:06.051 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cryptography-2.3-3.el8.x86_64 20/42 2021-03-09T08:46:06.069 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-1742.g89fb622 8/8 2021-03-09T08:46:06.230 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyOpenSSL-18.0.0-1.el8.noarch 21/42 2021-03-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-1742.g89fb622 8/8 2021-03-09T08:46:06.326 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-1742.g89fb622 1/8 2021-03-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-03-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-03-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-03-09T08:46:06.327 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-03-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-03-09T08:46:06.328 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-03-09T08:46:06.457 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 22/42 2021-03-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-03-09T08:46:06.596 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:46:06.751 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 23/42 2021-03-09T08:46:06.950 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/42 2021-03-09T08:46:07.218 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 25/42 2021-03-09T08:46:07.220 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:07.221 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:46:07.221 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-diskprediction-local-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:07.221 INFO:teuthology.orchestra.run.smithi099.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-03-09T08:46:07.221 INFO:teuthology.orchestra.run.smithi099.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-03-09T08:46:07.221 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-0.3.3-5.el8.x86_64 2021-03-09T08:46:07.222 INFO:teuthology.orchestra.run.smithi099.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-03-09T08:46:07.222 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-03-09T08:46:07.222 INFO:teuthology.orchestra.run.smithi099.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-03-09T08:46:07.222 INFO:teuthology.orchestra.run.smithi099.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-03-09T08:46:07.223 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:07.223 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:07.435 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 26/42 2021-03-09T08:46:07.815 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 27/42 2021-03-09T08:46:08.014 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 28/42 2021-03-09T08:46:08.140 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-rook 2021-03-09T08:46:08.211 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 29/42 2021-03-09T08:46:08.399 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 30/42 2021-03-09T08:46:08.560 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 31/42 2021-03-09T08:46:08.677 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:39 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:08.839 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-portend-2.6-1.el8.noarch 32/42 2021-03-09T08:46:09.101 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 33/42 2021-03-09T08:46:09.330 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:09.331 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2021-03-09T08:46:09.331 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:46:09.332 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2021-03-09T08:46:09.332 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:46:09.332 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 26 k 2021-03-09T08:46:09.332 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2021-03-09T08:46:09.333 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-03-09T08:46:09.333 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-03-09T08:46:09.333 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-03-09T08:46:09.333 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-03-09T08:46:09.333 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-03-09T08:46:09.334 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-03-09T08:46:09.334 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-03-09T08:46:09.334 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-03-09T08:46:09.335 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-03-09T08:46:09.335 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-03-09T08:46:09.336 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-03-09T08:46:09.336 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-03-09T08:46:09.336 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:09.337 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:46:09.337 INFO:teuthology.orchestra.run.smithi099.stdout:======================================================================================== 2021-03-09T08:46:09.337 INFO:teuthology.orchestra.run.smithi099.stdout:Install 13 Packages 2021-03-09T08:46:09.337 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:09.339 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 1.9 M 2021-03-09T08:46:09.339 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 24 M 2021-03-09T08:46:09.339 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:46:09.344 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 34/42 2021-03-09T08:46:09.368 INFO:teuthology.orchestra.run.smithi099.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 807 kB/s | 27 kB 00:00 2021-03-09T08:46:09.370 INFO:teuthology.orchestra.run.smithi099.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 562 kB/s | 20 kB 00:00 2021-03-09T08:46:09.372 INFO:teuthology.orchestra.run.smithi099.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 43 MB/s | 126 kB 00:00 2021-03-09T08:46:09.373 INFO:teuthology.orchestra.run.smithi099.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 37 MB/s | 110 kB 00:00 2021-03-09T08:46:09.375 INFO:teuthology.orchestra.run.smithi099.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 45 MB/s | 155 kB 00:00 2021-03-09T08:46:09.376 INFO:teuthology.orchestra.run.smithi099.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 27 MB/s | 43 kB 00:00 2021-03-09T08:46:09.435 INFO:teuthology.orchestra.run.smithi099.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 557 kB/s | 33 kB 00:00 2021-03-09T08:46:09.443 INFO:teuthology.orchestra.run.smithi099.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 244 kB/s | 16 kB 00:00 2021-03-09T08:46:09.458 INFO:teuthology.orchestra.run.smithi099.stdout:(9/13): ceph-mgr-rook-17.0.0-1742.g89fb622b.el8 213 kB/s | 26 kB 00:00 2021-03-09T08:46:09.473 INFO:teuthology.orchestra.run.smithi099.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 2.0 MB/s | 81 kB 00:00 2021-03-09T08:46:09.494 INFO:teuthology.orchestra.run.smithi099.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.0 MB/s | 61 kB 00:00 2021-03-09T08:46:09.523 INFO:teuthology.orchestra.run.smithi099.stdout:(12/13): python3-rsa-3.4.2-16.el8.noarch.rpm 1.1 MB/s | 73 kB 00:00 2021-03-09T08:46:09.599 INFO:teuthology.orchestra.run.smithi099.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 7.4 MB/s | 1.2 MB 00:00 2021-03-09T08:46:09.601 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:09.601 INFO:teuthology.orchestra.run.smithi099.stdout:Total 7.2 MB/s | 1.9 MB 00:00 2021-03-09T08:46:09.606 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:46:09.621 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:46:09.621 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:46:09.696 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 35/42 2021-03-09T08:46:09.698 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:46:09.701 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:46:10.042 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 36/42 2021-03-09T08:46:10.192 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:46:10.431 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-1742.g89fb622b.el8. 37/42 2021-03-09T08:46:10.441 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-03-09T08:46:10.599 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 38/42 2021-03-09T08:46:10.701 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-03-09T08:46:10.939 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-03-09T08:46:11.134 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-03-09T08:46:11.317 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-03-09T08:46:11.551 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-03-09T08:46:11.870 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-03-09T08:46:12.015 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 38/42 2021-03-09T08:46:12.090 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-03-09T08:46:12.255 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-osd-2:17.0.0-1742.g89fb622b.el8.x86_64 39/42 2021-03-09T08:46:12.617 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-osd-2:17.0.0-1742.g89fb622b.el8.x86_64 39/42 2021-03-09T08:46:12.822 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mon-2:17.0.0-1742.g89fb622b.el8.x86_64 40/42 2021-03-09T08:46:13.041 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mon-2:17.0.0-1742.g89fb622b.el8.x86_64 40/42 2021-03-09T08:46:13.187 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-03-09T08:46:13.230 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mds-2:17.0.0-1742.g89fb622b.el8.x86_64 41/42 2021-03-09T08:46:13.250 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mds-2:17.0.0-1742.g89fb622b.el8.x86_64 41/42 2021-03-09T08:46:13.412 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-2:17.0.0-1742.g89fb622b.el8.x86_64 42/42 2021-03-09T08:46:13.564 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-03-09T08:46:13.768 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-03-09T08:46:13.781 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-2:17.0.0-1742.g89fb622b.el8.x86_64 42/42 2021-03-09T08:46:13.781 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-2:17.0.0-1742.g89fb622b.el8.x86_64 1/42 2021-03-09T08:46:13.781 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mds-2:17.0.0-1742.g89fb622b.el8.x86_64 2/42 2021-03-09T08:46:13.781 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 3/42 2021-03-09T08:46:13.782 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mon-2:17.0.0-1742.g89fb622b.el8.x86_64 4/42 2021-03-09T08:46:13.782 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-osd-2:17.0.0-1742.g89fb622b.el8.x86_64 5/42 2021-03-09T08:46:13.782 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-1742.g89fb622b.el8. 6/42 2021-03-09T08:46:13.782 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/42 2021-03-09T08:46:13.783 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/42 2021-03-09T08:46:13.783 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/42 2021-03-09T08:46:13.783 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/42 2021-03-09T08:46:13.783 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyOpenSSL-18.0.0-1.el8.noarch 11/42 2021-03-09T08:46:13.784 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/42 2021-03-09T08:46:13.784 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/42 2021-03-09T08:46:13.784 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/42 2021-03-09T08:46:13.784 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-asn1crypto-0.24.0-3.el8.noarch 15/42 2021-03-09T08:46:13.785 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 16/42 2021-03-09T08:46:13.785 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 17/42 2021-03-09T08:46:13.785 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cryptography-2.3-3.el8.x86_64 18/42 2021-03-09T08:46:13.786 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-idna-2.5-5.el8.noarch 19/42 2021-03-09T08:46:13.786 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 20/42 2021-03-09T08:46:13.786 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 21/42 2021-03-09T08:46:13.786 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 22/42 2021-03-09T08:46:13.787 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 23/42 2021-03-09T08:46:13.787 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 24/42 2021-03-09T08:46:13.787 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 25/42 2021-03-09T08:46:13.787 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 26/42 2021-03-09T08:46:13.790 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 27/42 2021-03-09T08:46:13.790 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 28/42 2021-03-09T08:46:13.790 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 29/42 2021-03-09T08:46:13.790 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 30/42 2021-03-09T08:46:13.791 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 31/42 2021-03-09T08:46:13.791 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 32/42 2021-03-09T08:46:13.791 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 33/42 2021-03-09T08:46:13.792 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-portend-2.6-1.el8.noarch 34/42 2021-03-09T08:46:13.792 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 35/42 2021-03-09T08:46:13.792 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 36/42 2021-03-09T08:46:13.792 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 37/42 2021-03-09T08:46:13.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 38/42 2021-03-09T08:46:13.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 39/42 2021-03-09T08:46:13.793 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 40/42 2021-03-09T08:46:13.794 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 41/42 2021-03-09T08:46:13.978 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-03-09T08:46:14.142 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-rook-2:17.0.0-1742.g89fb622b.el8.noarch 13/13 2021-03-09T08:46:14.378 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-1742.g89fb622b.el8.noarch 13/13 2021-03-09T08:46:14.378 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-rook-2:17.0.0-1742.g89fb622b.el8.noarch 1/13 2021-03-09T08:46:14.379 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-03-09T08:46:14.379 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-03-09T08:46:14.380 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-03-09T08:46:14.380 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-03-09T08:46:14.381 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-03-09T08:46:14.381 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-03-09T08:46:14.381 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-03-09T08:46:14.382 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-03-09T08:46:14.382 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-03-09T08:46:14.383 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-03-09T08:46:14.383 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-03-09T08:46:14.614 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 42/42 2021-03-09T08:46:14.614 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:46:14.693 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-03-09T08:46:14.693 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:46:14.845 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:14.846 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:46:14.846 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:14.846 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mds-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:14.847 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:14.847 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:14.847 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mon-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:14.847 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-osd-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:14.847 INFO:teuthology.orchestra.run.smithi195.stdout: python3-asn1crypto-0.24.0-3.el8.noarch 2021-03-09T08:46:14.848 INFO:teuthology.orchestra.run.smithi195.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-03-09T08:46:14.848 INFO:teuthology.orchestra.run.smithi195.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-03-09T08:46:14.848 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-03-09T08:46:14.848 INFO:teuthology.orchestra.run.smithi195.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-03-09T08:46:14.849 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-03-09T08:46:14.849 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-03-09T08:46:14.849 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cryptography-2.3-3.el8.x86_64 2021-03-09T08:46:14.850 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-03-09T08:46:14.850 INFO:teuthology.orchestra.run.smithi195.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-03-09T08:46:14.850 INFO:teuthology.orchestra.run.smithi195.stdout: python3-idna-2.5-5.el8.noarch 2021-03-09T08:46:14.851 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-6.2-6.el8.noarch 2021-03-09T08:46:14.851 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-03-09T08:46:14.851 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-03-09T08:46:14.851 INFO:teuthology.orchestra.run.smithi195.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-03-09T08:46:14.852 INFO:teuthology.orchestra.run.smithi195.stdout: python3-mako-1.0.6-13.el8.noarch 2021-03-09T08:46:14.852 INFO:teuthology.orchestra.run.smithi195.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-03-09T08:46:14.852 INFO:teuthology.orchestra.run.smithi195.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-03-09T08:46:14.853 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-03-09T08:46:14.853 INFO:teuthology.orchestra.run.smithi195.stdout: python3-portend-2.6-1.el8.noarch 2021-03-09T08:46:14.853 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyOpenSSL-18.0.0-1.el8.noarch 2021-03-09T08:46:14.854 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pycparser-2.14-14.el8.noarch 2021-03-09T08:46:14.854 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-03-09T08:46:14.854 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pytz-2017.2-9.el8.noarch 2021-03-09T08:46:14.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-03-09T08:46:14.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-03-09T08:46:14.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-03-09T08:46:14.855 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-03-09T08:46:14.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-03-09T08:46:14.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-03-09T08:46:14.856 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-03-09T08:46:14.857 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-03-09T08:46:14.857 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-03-09T08:46:14.857 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-03-09T08:46:14.857 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-03-09T08:46:14.858 INFO:teuthology.orchestra.run.smithi195.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-03-09T08:46:14.858 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:14.859 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:14.911 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:14.912 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:46:14.912 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-rook-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:14.912 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-03-09T08:46:14.912 INFO:teuthology.orchestra.run.smithi099.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-03-09T08:46:14.913 INFO:teuthology.orchestra.run.smithi099.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-03-09T08:46:14.913 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-03-09T08:46:14.913 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-03-09T08:46:14.913 INFO:teuthology.orchestra.run.smithi099.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-03-09T08:46:14.914 INFO:teuthology.orchestra.run.smithi099.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-03-09T08:46:14.914 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-03-09T08:46:14.914 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-03-09T08:46:14.914 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-03-09T08:46:14.915 INFO:teuthology.orchestra.run.smithi099.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-03-09T08:46:14.915 INFO:teuthology.orchestra.run.smithi099.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-03-09T08:46:14.915 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:14.915 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:15.017 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-mgr-cephadm 2021-03-09T08:46:15.051 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-base 2021-03-09T08:46:15.556 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:46 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:15.581 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:08 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:16.073 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-base-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:46:16.217 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:16.218 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:16.218 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:46:16.219 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:16.219 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:46:16.219 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 74 k 2021-03-09T08:46:16.219 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2021-03-09T08:46:16.220 INFO:teuthology.orchestra.run.smithi099.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-03-09T08:46:16.220 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-03-09T08:46:16.220 INFO:teuthology.orchestra.run.smithi099.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-03-09T08:46:16.220 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-03-09T08:46:16.220 INFO:teuthology.orchestra.run.smithi099.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-03-09T08:46:16.221 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:16.221 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:46:16.221 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:16.221 INFO:teuthology.orchestra.run.smithi099.stdout:Install 6 Packages 2021-03-09T08:46:16.222 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:16.222 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 5.8 M 2021-03-09T08:46:16.223 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 24 M 2021-03-09T08:46:16.223 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:46:16.241 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:16.243 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2021-03-09T08:46:16.243 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:16.266 INFO:teuthology.orchestra.run.smithi099.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 12 MB/s | 538 kB 00:00 2021-03-09T08:46:16.307 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install cephadm 2021-03-09T08:46:16.321 INFO:teuthology.orchestra.run.smithi099.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 48 MB/s | 4.8 MB 00:00 2021-03-09T08:46:16.322 INFO:teuthology.orchestra.run.smithi099.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 383 kB/s | 21 kB 00:00 2021-03-09T08:46:16.365 INFO:teuthology.orchestra.run.smithi099.stdout:(4/6): ceph-mgr-cephadm-17.0.0-1742.g89fb622b.e 513 kB/s | 74 kB 00:00 2021-03-09T08:46:16.386 INFO:teuthology.orchestra.run.smithi099.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 5.4 MB/s | 355 kB 00:00 2021-03-09T08:46:16.409 INFO:teuthology.orchestra.run.smithi099.stdout:(6/6): python3-remoto-1.1.4-4.el8.noarch.rpm 550 kB/s | 48 kB 00:00 2021-03-09T08:46:16.411 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:16.411 INFO:teuthology.orchestra.run.smithi099.stdout:Total 30 MB/s | 5.8 MB 00:00 2021-03-09T08:46:16.414 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:46:16.423 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:46:16.424 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:46:16.510 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:46:16.513 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:46:16.780 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:46:16.832 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:09 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:17.047 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-03-09T08:46:17.297 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-03-09T08:46:17.739 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:17.740 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:17.740 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:46:17.740 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:17.740 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:46:17.741 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 65 k 2021-03-09T08:46:17.741 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:17.741 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:46:17.741 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:17.741 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2021-03-09T08:46:17.742 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:17.742 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 65 k 2021-03-09T08:46:17.742 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 284 k 2021-03-09T08:46:17.742 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:46:17.885 INFO:teuthology.orchestra.run.smithi195.stdout:cephadm-17.0.0-1742.g89fb622b.el8.noarch.rpm 447 kB/s | 65 kB 00:00 2021-03-09T08:46:17.886 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:17.886 INFO:teuthology.orchestra.run.smithi195.stdout:Total 443 kB/s | 65 kB 00:00 2021-03-09T08:46:17.887 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:46:17.894 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:46:17.894 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:46:17.895 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:46:17.899 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:46:17.945 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:46:18.294 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-03-09T08:46:18.915 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-03-09T08:46:19.156 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-03-09T08:46:19.341 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 6/6 2021-03-09T08:46:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 6/6 2021-03-09T08:46:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/6 2021-03-09T08:46:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-03-09T08:46:19.605 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-03-09T08:46:19.606 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-03-09T08:46:19.606 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-03-09T08:46:19.875 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-03-09T08:46:19.875 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:46:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/1 2021-03-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:20.127 INFO:teuthology.orchestra.run.smithi099.stdout: python3-apipkg-1.5-6.el8.noarch 2021-03-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel-2.5.1-5.el8.noarch 2021-03-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi099.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-03-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-03-09T08:46:20.128 INFO:teuthology.orchestra.run.smithi099.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-03-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:20.129 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:20.227 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install ceph-fuse 2021-03-09T08:46:20.241 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/1 2021-03-09T08:46:20.421 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/1 2021-03-09T08:46:20.674 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/1 2021-03-09T08:46:20.675 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:46:20.765 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:51 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:20.893 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:20.893 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:46:20.893 INFO:teuthology.orchestra.run.smithi195.stdout: cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:20.893 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:20.894 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:21.054 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-immutable-object-cache 2021-03-09T08:46:21.425 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:21.426 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:21.426 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repository Size 2021-03-09T08:46:21.426 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:21.427 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:46:21.427 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 826 k 2021-03-09T08:46:21.427 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2021-03-09T08:46:21.427 INFO:teuthology.orchestra.run.smithi099.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-03-09T08:46:21.427 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:21.428 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:46:21.428 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:21.428 INFO:teuthology.orchestra.run.smithi099.stdout:Install 2 Packages 2021-03-09T08:46:21.428 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:21.429 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 908 k 2021-03-09T08:46:21.429 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 3.2 M 2021-03-09T08:46:21.429 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:46:21.460 INFO:teuthology.orchestra.run.smithi099.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.4 MB/s | 82 kB 00:00 2021-03-09T08:46:21.596 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:14 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:21.644 INFO:teuthology.orchestra.run.smithi099.stdout:(2/2): ceph-fuse-17.0.0-1742.g89fb622b.el8.x86_ 3.7 MB/s | 826 kB 00:00 2021-03-09T08:46:21.645 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:21.646 INFO:teuthology.orchestra.run.smithi099.stdout:Total 4.0 MB/s | 908 kB 00:00 2021-03-09T08:46:21.647 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:46:21.668 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:46:21.669 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:46:21.745 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:46:21.748 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:46:21.912 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:46:22.223 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-03-09T08:46:22.268 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:22.269 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:22.270 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2021-03-09T08:46:22.270 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:22.270 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:46:22.270 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 155 k 2021-03-09T08:46:22.271 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:22.271 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:46:22.271 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:22.271 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2021-03-09T08:46:22.271 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:22.272 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 155 k 2021-03-09T08:46:22.272 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 587 k 2021-03-09T08:46:22.272 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:46:22.415 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : ceph-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 2/2 2021-03-09T08:46:22.438 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-immutable-object-cache-17.0.0-1742.g89fb62 926 kB/s | 155 kB 00:00 2021-03-09T08:46:22.438 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:22.439 INFO:teuthology.orchestra.run.smithi195.stdout:Total 919 kB/s | 155 kB 00:00 2021-03-09T08:46:22.439 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:46:22.452 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:46:22.452 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:46:22.507 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:46:22.508 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:46:22.753 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: ceph-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 2/2 2021-03-09T08:46:22.754 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : ceph-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 1/2 2021-03-09T08:46:23.007 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-03-09T08:46:23.007 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:46:23.237 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:46:23.310 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:23.311 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:46:23.311 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-03-09T08:46:23.311 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:23.312 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:23.472 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados-devel 2021-03-09T08:46:24.019 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:55 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:24.665 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:24.666 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:24.667 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2021-03-09T08:46:24.667 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:24.667 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:46:24.667 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 125 k 2021-03-09T08:46:24.668 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:24.668 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:46:24.668 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:24.668 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2021-03-09T08:46:24.668 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:24.669 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 125 k 2021-03-09T08:46:24.669 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 439 k 2021-03-09T08:46:24.669 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:46:24.761 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-1742.g89fb622b. 1/1 2021-03-09T08:46:24.835 INFO:teuthology.orchestra.run.smithi099.stdout:librados-devel-17.0.0-1742.g89fb622b.el8.x86_64 743 kB/s | 125 kB 00:00 2021-03-09T08:46:24.836 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:24.836 INFO:teuthology.orchestra.run.smithi099.stdout:Total 738 kB/s | 125 kB 00:00 2021-03-09T08:46:24.837 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:46:24.843 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:46:24.843 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:46:24.897 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:46:24.900 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:46:25.030 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:46:25.126 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-1742.g89fb622b. 1/1 2021-03-09T08:46:25.264 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : librados-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:25.367 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-1742.g89fb622b. 1/1 2021-03-09T08:46:25.367 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:46:25.503 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: librados-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:25.633 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:25.634 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:46:25.634 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-immutable-object-cache-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:25.634 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:25.634 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:25.752 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : librados-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:25.752 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:46:25.789 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr 2021-03-09T08:46:25.983 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:25.984 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:46:25.984 INFO:teuthology.orchestra.run.smithi099.stdout: librados-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:25.984 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:25.984 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:26.150 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs2 2021-03-09T08:46:26.330 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:19 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:26.692 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:57 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:26.821 INFO:teuthology.orchestra.run.smithi195.stdout:Package ceph-mgr-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:46:26.985 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:26.986 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2021-03-09T08:46:26.987 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:27.066 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-dashboard 2021-03-09T08:46:27.189 INFO:teuthology.orchestra.run.smithi099.stdout:Package libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:46:27.354 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:27.355 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2021-03-09T08:46:27.355 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:27.423 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install libcephfs-devel 2021-03-09T08:46:27.590 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:20 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:27.958 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:01:58 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:28.244 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:28.245 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:28.245 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:46:28.246 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:28.246 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:46:28.246 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 2.3 M 2021-03-09T08:46:28.246 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2021-03-09T08:46:28.247 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 23 k 2021-03-09T08:46:28.247 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 8.9 k 2021-03-09T08:46:28.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-03-09T08:46:28.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-03-09T08:46:28.247 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-03-09T08:46:28.248 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:28.248 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:46:28.248 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:28.248 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2021-03-09T08:46:28.249 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:28.249 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 2.5 M 2021-03-09T08:46:28.250 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 15 M 2021-03-09T08:46:28.250 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:46:28.351 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): ceph-prometheus-alerts-17.0.0-1742.g89fb 86 kB/s | 8.9 kB 00:00 2021-03-09T08:46:28.382 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): ceph-grafana-dashboards-17.0.0-1742.g89f 170 kB/s | 23 kB 00:00 2021-03-09T08:46:28.383 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2021-03-09T08:46:28.441 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 568 kB/s | 34 kB 00:00 2021-03-09T08:46:28.499 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.7 MB/s | 196 kB 00:00 2021-03-09T08:46:28.528 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): ceph-mgr-dashboard-17.0.0-1742.g89fb622b 8.0 MB/s | 2.3 MB 00:00 2021-03-09T08:46:28.530 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:28.531 INFO:teuthology.orchestra.run.smithi195.stdout:Total 9.0 MB/s | 2.5 MB 00:00 2021-03-09T08:46:28.532 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:46:28.541 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:46:28.541 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:46:28.620 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:28.621 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:28.622 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2021-03-09T08:46:28.622 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:28.622 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:46:28.622 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 23 k 2021-03-09T08:46:28.623 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:28.623 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:46:28.623 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:28.623 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2021-03-09T08:46:28.624 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:28.624 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 23 k 2021-03-09T08:46:28.624 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 89 k 2021-03-09T08:46:28.624 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:46:28.739 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:46:28.742 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:46:28.745 INFO:teuthology.orchestra.run.smithi099.stdout:libcephfs-devel-17.0.0-1742.g89fb622b.el8.x86_6 186 kB/s | 23 kB 00:00 2021-03-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:28.746 INFO:teuthology.orchestra.run.smithi099.stdout:Total 184 kB/s | 23 kB 00:00 2021-03-09T08:46:28.747 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:46:28.751 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:46:28.753 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:46:28.756 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:46:28.806 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:46:28.996 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : libcephfs-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:29.020 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:46:29.198 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:29.251 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-03-09T08:46:29.471 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-03-09T08:46:29.512 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : libcephfs-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:29.513 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:46:29.679 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-03-09T08:46:29.740 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:29.741 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:46:29.741 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:29.741 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:29.742 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:29.792 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-1742.g89fb622b.el8.n 4/6 2021-03-09T08:46:29.815 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librados2 2021-03-09T08:46:30.350 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:01 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:30.803 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-1742.g89fb622b.el8. 5/6 2021-03-09T08:46:30.822 INFO:teuthology.orchestra.run.smithi099.stdout:Package librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:46:30.986 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:30.986 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2021-03-09T08:46:30.987 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:31.054 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install librbd1 2021-03-09T08:46:31.085 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-1742.g89fb622b.el8.noarc 6/6 2021-03-09T08:46:31.322 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-1742.g89fb622b.el8.noarc 6/6 2021-03-09T08:46:31.323 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-1742.g89fb622b.el8. 1/6 2021-03-09T08:46:31.323 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-1742.g89fb622b.el8.noarc 2/6 2021-03-09T08:46:31.323 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-1742.g89fb622b.el8.n 3/6 2021-03-09T08:46:31.323 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-03-09T08:46:31.323 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-03-09T08:46:31.591 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:02 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:31.607 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-03-09T08:46:31.607 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:46:31.816 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:31.816 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:46:31.817 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-grafana-dashboards-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:31.817 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:31.817 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-prometheus-alerts-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:31.817 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-03-09T08:46:31.817 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-03-09T08:46:31.818 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes-2.4.1-12.el8.noarch 2021-03-09T08:46:31.818 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:31.818 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:31.880 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-diskprediction-local 2021-03-09T08:46:32.088 INFO:teuthology.orchestra.run.smithi099.stdout:Package librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:46:32.251 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2021-03-09T08:46:32.252 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:32.323 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rados 2021-03-09T08:46:32.412 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:25 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:32.865 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:03 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:33.095 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:33.096 INFO:teuthology.orchestra.run.smithi195.stdout:===================================================================================================== 2021-03-09T08:46:33.096 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:46:33.096 INFO:teuthology.orchestra.run.smithi195.stdout:===================================================================================================== 2021-03-09T08:46:33.097 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:46:33.097 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 7.4 M 2021-03-09T08:46:33.097 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2021-03-09T08:46:33.097 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-03-09T08:46:33.097 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-03-09T08:46:33.098 INFO:teuthology.orchestra.run.smithi195.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-03-09T08:46:33.098 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-03-09T08:46:33.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-03-09T08:46:33.098 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-03-09T08:46:33.099 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-03-09T08:46:33.099 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:33.099 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:46:33.099 INFO:teuthology.orchestra.run.smithi195.stdout:===================================================================================================== 2021-03-09T08:46:33.099 INFO:teuthology.orchestra.run.smithi195.stdout:Install 8 Packages 2021-03-09T08:46:33.100 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:33.100 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 35 M 2021-03-09T08:46:33.101 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 194 M 2021-03-09T08:46:33.101 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:46:33.201 INFO:teuthology.orchestra.run.smithi195.stdout:(1/8): openblas-0.3.3-5.el8.x86_64.rpm 42 MB/s | 4.3 MB 00:00 2021-03-09T08:46:33.264 INFO:teuthology.orchestra.run.smithi195.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 59 MB/s | 3.7 MB 00:00 2021-03-09T08:46:33.270 INFO:teuthology.orchestra.run.smithi195.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 37 MB/s | 225 kB 00:00 2021-03-09T08:46:33.282 INFO:teuthology.orchestra.run.smithi195.stdout:(4/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 24 MB/s | 4.3 MB 00:00 2021-03-09T08:46:33.306 INFO:teuthology.orchestra.run.smithi195.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 27 MB/s | 639 kB 00:00 2021-03-09T08:46:33.316 INFO:teuthology.orchestra.run.smithi195.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 16 MB/s | 169 kB 00:00 2021-03-09T08:46:33.363 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rados-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:46:33.534 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:33.535 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2021-03-09T08:46:33.536 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:33.558 INFO:teuthology.orchestra.run.smithi195.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 50 MB/s | 15 MB 00:00 2021-03-09T08:46:33.591 INFO:teuthology.orchestra.run.smithi195.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-174 15 MB/s | 7.4 MB 00:00 2021-03-09T08:46:33.594 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:33.594 INFO:teuthology.orchestra.run.smithi195.stdout:Total 71 MB/s | 35 MB 00:00 2021-03-09T08:46:33.603 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:46:33.607 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rgw 2021-03-09T08:46:33.623 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:46:33.623 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:46:33.937 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:46:33.941 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:46:34.147 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:05 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:34.523 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:46:34.692 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-03-09T08:46:34.836 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-03-09T08:46:34.997 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rgw-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:46:35.037 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-03-09T08:46:35.165 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:35.166 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2021-03-09T08:46:35.166 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:35.513 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-cephfs 2021-03-09T08:46:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-03-09T08:46:35.692 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-03-09T08:46:36.054 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:07 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:36.191 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-03-09T08:46:36.392 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-03-09T08:46:36.546 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-cephfs-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:46:36.713 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:36.714 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2021-03-09T08:46:36.715 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:36.783 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install python3-rbd 2021-03-09T08:46:37.112 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-03-09T08:46:37.322 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:08 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:37.476 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-03-09T08:46:37.700 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-03-09T08:46:37.803 INFO:teuthology.orchestra.run.smithi099.stdout:Package python3-rbd-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:46:37.970 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:37.971 INFO:teuthology.orchestra.run.smithi099.stdout:Nothing to do. 2021-03-09T08:46:37.972 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:38.040 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-fuse 2021-03-09T08:46:38.576 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:09 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:39.229 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:39.230 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:39.230 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2021-03-09T08:46:39.230 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:39.231 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:46:39.231 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 86 k 2021-03-09T08:46:39.231 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:39.231 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:46:39.232 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:39.232 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2021-03-09T08:46:39.232 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:39.232 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 86 k 2021-03-09T08:46:39.232 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 298 k 2021-03-09T08:46:39.233 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:46:39.376 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-fuse-17.0.0-1742.g89fb622b.el8.x86_64.rpm 590 kB/s | 86 kB 00:00 2021-03-09T08:46:39.377 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:39.377 INFO:teuthology.orchestra.run.smithi099.stdout:Total 585 kB/s | 86 kB 00:00 2021-03-09T08:46:39.377 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:46:39.389 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:46:39.389 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:46:39.441 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:46:39.444 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:46:39.551 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:46:39.710 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-03-09T08:46:39.752 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:39.991 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:40.263 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:40.264 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:46:40.520 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:40.520 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:46:40.520 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:40.521 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:40.521 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:40.699 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-mirror 2021-03-09T08:46:40.986 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-03-09T08:46:41.191 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-1742.g89fb622 8/8 2021-03-09T08:46:41.354 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:12 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:41.450 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-1742.g89fb622 8/8 2021-03-09T08:46:41.450 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-1742.g89fb622 1/8 2021-03-09T08:46:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-03-09T08:46:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-03-09T08:46:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-03-09T08:46:41.451 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-03-09T08:46:41.452 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-03-09T08:46:41.452 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-03-09T08:46:41.728 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-03-09T08:46:41.729 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:46:41.973 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:41.973 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:46:41.973 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-diskprediction-local-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:41.974 INFO:teuthology.orchestra.run.smithi195.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-03-09T08:46:41.974 INFO:teuthology.orchestra.run.smithi195.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-03-09T08:46:41.974 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-0.3.3-5.el8.x86_64 2021-03-09T08:46:41.974 INFO:teuthology.orchestra.run.smithi195.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-03-09T08:46:41.975 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-03-09T08:46:41.975 INFO:teuthology.orchestra.run.smithi195.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-03-09T08:46:41.975 INFO:teuthology.orchestra.run.smithi195.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-03-09T08:46:41.975 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:41.976 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:42.019 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:42.020 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:42.020 INFO:teuthology.orchestra.run.smithi099.stdout: Package Arch Version Repo Size 2021-03-09T08:46:42.020 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:42.020 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:46:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.4 M 2021-03-09T08:46:42.021 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:42.021 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:46:42.021 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:42.022 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2021-03-09T08:46:42.022 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:42.022 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 3.4 M 2021-03-09T08:46:42.022 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 18 M 2021-03-09T08:46:42.022 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:46:42.065 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-rook 2021-03-09T08:46:42.341 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-mirror-17.0.0-1742.g89fb622b.el8.x86_64.rpm 10 MB/s | 3.4 MB 00:00 2021-03-09T08:46:42.342 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:42.343 INFO:teuthology.orchestra.run.smithi099.stdout:Total 10 MB/s | 3.4 MB 00:00 2021-03-09T08:46:42.343 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:46:42.356 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:46:42.357 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:46:42.418 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:46:42.421 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:46:42.593 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:35 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:42.865 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:46:43.097 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-mirror-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:43.309 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:43.310 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2021-03-09T08:46:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:46:43.311 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2021-03-09T08:46:43.311 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:46:43.311 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 26 k 2021-03-09T08:46:43.311 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2021-03-09T08:46:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-03-09T08:46:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-03-09T08:46:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-03-09T08:46:43.312 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-03-09T08:46:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-03-09T08:46:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-03-09T08:46:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-03-09T08:46:43.313 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-03-09T08:46:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-03-09T08:46:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-03-09T08:46:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-03-09T08:46:43.314 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-03-09T08:46:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:43.315 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:46:43.315 INFO:teuthology.orchestra.run.smithi195.stdout:======================================================================================== 2021-03-09T08:46:43.315 INFO:teuthology.orchestra.run.smithi195.stdout:Install 13 Packages 2021-03-09T08:46:43.315 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:43.316 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 1.9 M 2021-03-09T08:46:43.316 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 24 M 2021-03-09T08:46:43.317 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:46:43.351 INFO:teuthology.orchestra.run.smithi195.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 717 kB/s | 27 kB 00:00 2021-03-09T08:46:43.353 INFO:teuthology.orchestra.run.smithi195.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 502 kB/s | 20 kB 00:00 2021-03-09T08:46:43.356 INFO:teuthology.orchestra.run.smithi195.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 21 MB/s | 126 kB 00:00 2021-03-09T08:46:43.357 INFO:teuthology.orchestra.run.smithi195.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 30 MB/s | 110 kB 00:00 2021-03-09T08:46:43.360 INFO:teuthology.orchestra.run.smithi195.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 16 MB/s | 43 kB 00:00 2021-03-09T08:46:43.368 INFO:teuthology.orchestra.run.smithi195.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 14 MB/s | 155 kB 00:00 2021-03-09T08:46:43.416 INFO:teuthology.orchestra.run.smithi195.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 338 kB/s | 16 kB 00:00 2021-03-09T08:46:43.420 INFO:teuthology.orchestra.run.smithi195.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 554 kB/s | 33 kB 00:00 2021-03-09T08:46:43.435 INFO:teuthology.orchestra.run.smithi195.stdout:(9/13): ceph-mgr-rook-17.0.0-1742.g89fb622b.el8 213 kB/s | 26 kB 00:00 2021-03-09T08:46:43.465 INFO:teuthology.orchestra.run.smithi195.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.7 MB/s | 81 kB 00:00 2021-03-09T08:46:43.474 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-mirror-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:43.481 INFO:teuthology.orchestra.run.smithi195.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.6 MB/s | 61 kB 00:00 2021-03-09T08:46:43.500 INFO:teuthology.orchestra.run.smithi195.stdout:(12/13): python3-rsa-3.4.2-16.el8.noarch.rpm 1.1 MB/s | 73 kB 00:00 2021-03-09T08:46:43.558 INFO:teuthology.orchestra.run.smithi195.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 8.4 MB/s | 1.2 MB 00:00 2021-03-09T08:46:43.560 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:43.560 INFO:teuthology.orchestra.run.smithi195.stdout:Total 7.7 MB/s | 1.9 MB 00:00 2021-03-09T08:46:43.565 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:46:43.579 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:46:43.580 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:46:43.660 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:46:43.663 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:46:43.688 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-mirror-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:43.688 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:46:43.929 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:43.930 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:46:43.930 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-mirror-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:43.930 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:43.931 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:44.105 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum -y install rbd-nbd 2021-03-09T08:46:44.154 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:46:44.413 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-03-09T08:46:44.648 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-03-09T08:46:44.653 INFO:teuthology.orchestra.run.smithi099.stdout:Last metadata expiration check: 0:02:15 ago on Tue 09 Mar 2021 08:44:29 AM UTC. 2021-03-09T08:46:44.918 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-03-09T08:46:45.148 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-03-09T08:46:45.330 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2021-03-09T08:46:45.331 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:45.331 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2021-03-09T08:46:45.332 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:45.332 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2021-03-09T08:46:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 165 k 2021-03-09T08:46:45.332 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:45.333 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2021-03-09T08:46:45.333 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2021-03-09T08:46:45.333 INFO:teuthology.orchestra.run.smithi099.stdout:Install 1 Package 2021-03-09T08:46:45.333 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:45.333 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 165 k 2021-03-09T08:46:45.334 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 649 k 2021-03-09T08:46:45.334 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2021-03-09T08:46:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-03-09T08:46:45.504 INFO:teuthology.orchestra.run.smithi099.stdout:rbd-nbd-17.0.0-1742.g89fb622b.el8.x86_64.rpm 964 kB/s | 165 kB 00:00 2021-03-09T08:46:45.505 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:45.505 INFO:teuthology.orchestra.run.smithi099.stdout:Total 956 kB/s | 165 kB 00:00 2021-03-09T08:46:45.505 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2021-03-09T08:46:45.517 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2021-03-09T08:46:45.518 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2021-03-09T08:46:45.568 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2021-03-09T08:46:45.571 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2021-03-09T08:46:45.691 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2021-03-09T08:46:45.942 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : rbd-nbd-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:46.072 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-03-09T08:46:46.182 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: rbd-nbd-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:46.427 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : rbd-nbd-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:46:46.428 INFO:teuthology.orchestra.run.smithi099.stdout:Installed products updated. 2021-03-09T08:46:46.632 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-03-09T08:46:46.860 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:46.860 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2021-03-09T08:46:46.860 INFO:teuthology.orchestra.run.smithi099.stdout: rbd-nbd-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:46:46.861 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:46:46.861 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2021-03-09T08:46:46.909 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-03-09T08:46:47.385 DEBUG:teuthology.parallel:result is None 2021-03-09T08:46:48.056 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-03-09T08:46:48.366 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-03-09T08:46:48.590 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-03-09T08:46:48.817 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-03-09T08:46:48.957 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-rook-2:17.0.0-1742.g89fb622b.el8.noarch 13/13 2021-03-09T08:46:49.237 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-1742.g89fb622b.el8.noarch 13/13 2021-03-09T08:46:49.237 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-rook-2:17.0.0-1742.g89fb622b.el8.noarch 1/13 2021-03-09T08:46:49.237 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-03-09T08:46:49.238 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-03-09T08:46:49.238 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-03-09T08:46:49.238 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-03-09T08:46:49.239 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-03-09T08:46:49.239 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-03-09T08:46:49.240 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-03-09T08:46:49.240 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-03-09T08:46:49.240 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-03-09T08:46:49.240 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-03-09T08:46:49.241 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-03-09T08:46:49.622 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-03-09T08:46:49.622 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:46:49.830 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:49.831 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:46:49.831 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-rook-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:49.831 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-03-09T08:46:49.832 INFO:teuthology.orchestra.run.smithi195.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-03-09T08:46:49.832 INFO:teuthology.orchestra.run.smithi195.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-03-09T08:46:49.832 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-03-09T08:46:49.832 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-03-09T08:46:49.832 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-03-09T08:46:49.833 INFO:teuthology.orchestra.run.smithi195.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-03-09T08:46:49.833 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-03-09T08:46:49.833 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-03-09T08:46:49.833 INFO:teuthology.orchestra.run.smithi195.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-03-09T08:46:49.834 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-03-09T08:46:49.834 INFO:teuthology.orchestra.run.smithi195.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-03-09T08:46:49.834 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:49.834 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:49.906 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-mgr-cephadm 2021-03-09T08:46:50.444 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:43 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:51.110 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:51.110 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:51.111 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:46:51.111 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:51.111 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:46:51.111 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm noarch 2:17.0.0-1742.g89fb622b.el8 ceph-noarch 74 k 2021-03-09T08:46:51.112 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2021-03-09T08:46:51.112 INFO:teuthology.orchestra.run.smithi195.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-03-09T08:46:51.112 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-03-09T08:46:51.112 INFO:teuthology.orchestra.run.smithi195.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-03-09T08:46:51.112 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-03-09T08:46:51.113 INFO:teuthology.orchestra.run.smithi195.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-03-09T08:46:51.113 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:51.113 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:46:51.113 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:51.113 INFO:teuthology.orchestra.run.smithi195.stdout:Install 6 Packages 2021-03-09T08:46:51.114 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:51.115 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 5.8 M 2021-03-09T08:46:51.115 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 24 M 2021-03-09T08:46:51.115 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:46:51.158 INFO:teuthology.orchestra.run.smithi195.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 12 MB/s | 538 kB 00:00 2021-03-09T08:46:51.207 INFO:teuthology.orchestra.run.smithi195.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 428 kB/s | 21 kB 00:00 2021-03-09T08:46:51.227 INFO:teuthology.orchestra.run.smithi195.stdout:(3/6): python3-babel-2.5.1-5.el8.noarch.rpm 42 MB/s | 4.8 MB 00:00 2021-03-09T08:46:51.257 INFO:teuthology.orchestra.run.smithi195.stdout:(4/6): ceph-mgr-cephadm-17.0.0-1742.g89fb622b.e 513 kB/s | 74 kB 00:00 2021-03-09T08:46:51.272 INFO:teuthology.orchestra.run.smithi195.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 5.4 MB/s | 355 kB 00:00 2021-03-09T08:46:51.305 INFO:teuthology.orchestra.run.smithi195.stdout:(6/6): python3-remoto-1.1.4-4.el8.noarch.rpm 611 kB/s | 48 kB 00:00 2021-03-09T08:46:51.308 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:51.308 INFO:teuthology.orchestra.run.smithi195.stdout:Total 30 MB/s | 5.8 MB 00:00 2021-03-09T08:46:51.310 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:46:51.320 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:46:51.320 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:46:51.413 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:46:51.416 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:46:51.685 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:46:52.538 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-03-09T08:46:52.808 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-03-09T08:46:53.800 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-03-09T08:46:54.143 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-03-09T08:46:54.361 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-03-09T08:46:54.591 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 6/6 2021-03-09T08:46:54.849 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 6/6 2021-03-09T08:46:54.849 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 1/6 2021-03-09T08:46:54.850 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-03-09T08:46:54.850 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-03-09T08:46:54.850 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-03-09T08:46:54.850 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-03-09T08:46:55.124 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-03-09T08:46:55.124 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:46:55.346 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:55.347 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:46:55.347 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-cephadm-2:17.0.0-1742.g89fb622b.el8.noarch 2021-03-09T08:46:55.347 INFO:teuthology.orchestra.run.smithi195.stdout: python3-apipkg-1.5-6.el8.noarch 2021-03-09T08:46:55.347 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel-2.5.1-5.el8.noarch 2021-03-09T08:46:55.348 INFO:teuthology.orchestra.run.smithi195.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-03-09T08:46:55.348 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-03-09T08:46:55.348 INFO:teuthology.orchestra.run.smithi195.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-03-09T08:46:55.348 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:55.349 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:55.429 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install ceph-fuse 2021-03-09T08:46:55.973 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:48 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:56.648 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:56.649 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repository Size 2021-03-09T08:46:56.649 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:56.650 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:46:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 826 k 2021-03-09T08:46:56.650 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2021-03-09T08:46:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-03-09T08:46:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:56.651 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:46:56.651 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:56.651 INFO:teuthology.orchestra.run.smithi195.stdout:Install 2 Packages 2021-03-09T08:46:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:56.652 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 908 k 2021-03-09T08:46:56.652 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 3.2 M 2021-03-09T08:46:56.652 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:46:56.683 INFO:teuthology.orchestra.run.smithi195.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.4 MB/s | 82 kB 00:00 2021-03-09T08:46:56.867 INFO:teuthology.orchestra.run.smithi195.stdout:(2/2): ceph-fuse-17.0.0-1742.g89fb622b.el8.x86_ 3.7 MB/s | 826 kB 00:00 2021-03-09T08:46:56.869 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:46:56.869 INFO:teuthology.orchestra.run.smithi195.stdout:Total 4.0 MB/s | 908 kB 00:00 2021-03-09T08:46:56.870 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:46:56.892 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:46:56.892 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:46:56.967 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:46:56.970 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:46:57.165 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:46:57.471 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-03-09T08:46:57.655 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : ceph-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 2/2 2021-03-09T08:46:57.979 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: ceph-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 2/2 2021-03-09T08:46:57.979 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : ceph-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 1/2 2021-03-09T08:46:58.248 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-03-09T08:46:58.249 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:46:58.528 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:58.528 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:46:58.529 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-03-09T08:46:58.529 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:58.529 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:46:58.715 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados-devel 2021-03-09T08:46:59.263 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:52 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:46:59.929 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:46:59.930 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:59.930 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2021-03-09T08:46:59.931 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:59.931 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:46:59.931 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 125 k 2021-03-09T08:46:59.931 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:59.931 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:46:59.932 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:46:59.932 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2021-03-09T08:46:59.932 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:46:59.932 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 125 k 2021-03-09T08:46:59.933 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 439 k 2021-03-09T08:46:59.933 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:47:00.098 INFO:teuthology.orchestra.run.smithi195.stdout:librados-devel-17.0.0-1742.g89fb622b.el8.x86_64 746 kB/s | 125 kB 00:00 2021-03-09T08:47:00.099 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:47:00.099 INFO:teuthology.orchestra.run.smithi195.stdout:Total 741 kB/s | 125 kB 00:00 2021-03-09T08:47:00.099 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:47:00.106 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:47:00.107 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:47:00.173 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:47:00.176 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:47:00.313 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:47:00.487 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : librados-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:00.744 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: librados-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:01.045 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : librados-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:01.045 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:47:01.311 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:01.312 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:47:01.312 INFO:teuthology.orchestra.run.smithi195.stdout: librados-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:47:01.312 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:01.313 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:01.512 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs2 2021-03-09T08:47:02.056 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:55 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:02.724 INFO:teuthology.orchestra.run.smithi195.stdout:Package libcephfs2-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:47:02.889 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:02.890 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2021-03-09T08:47:02.890 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:02.954 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install libcephfs-devel 2021-03-09T08:47:03.490 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:56 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:04.156 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:04.157 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:04.157 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2021-03-09T08:47:04.157 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:04.158 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:47:04.158 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 23 k 2021-03-09T08:47:04.158 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:04.158 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:47:04.158 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:04.159 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2021-03-09T08:47:04.159 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:04.159 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 23 k 2021-03-09T08:47:04.159 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 89 k 2021-03-09T08:47:04.159 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:47:04.281 INFO:teuthology.orchestra.run.smithi195.stdout:libcephfs-devel-17.0.0-1742.g89fb622b.el8.x86_6 187 kB/s | 23 kB 00:00 2021-03-09T08:47:04.281 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:47:04.282 INFO:teuthology.orchestra.run.smithi195.stdout:Total 185 kB/s | 23 kB 00:00 2021-03-09T08:47:04.282 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:47:04.286 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:47:04.286 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:47:04.288 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:47:04.292 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:47:04.345 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:47:04.528 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : libcephfs-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:04.743 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:04.963 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : libcephfs-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:04.963 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:47:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:05.186 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:47:05.186 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-devel-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:47:05.187 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:05.187 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:05.265 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librados2 2021-03-09T08:47:05.804 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:01:58 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:06.299 INFO:teuthology.orchestra.run.smithi195.stdout:Package librados2-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:47:06.464 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:06.465 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2021-03-09T08:47:06.465 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:06.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install librbd1 2021-03-09T08:47:07.083 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:00 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:07.875 INFO:teuthology.orchestra.run.smithi195.stdout:Package librbd1-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:47:08.042 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:08.043 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2021-03-09T08:47:08.044 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:08.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rados 2021-03-09T08:47:08.748 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:01 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:09.248 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rados-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:47:09.415 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:09.416 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2021-03-09T08:47:09.416 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:09.490 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rgw 2021-03-09T08:47:10.028 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:03 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:10.556 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rgw-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:47:10.722 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:10.723 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2021-03-09T08:47:10.723 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:10.798 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-cephfs 2021-03-09T08:47:11.342 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:04 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:11.839 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-cephfs-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:47:12.006 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:12.007 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2021-03-09T08:47:12.007 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:12.081 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install python3-rbd 2021-03-09T08:47:12.617 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:05 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:13.131 INFO:teuthology.orchestra.run.smithi195.stdout:Package python3-rbd-2:17.0.0-1742.g89fb622b.el8.x86_64 is already installed. 2021-03-09T08:47:13.297 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:13.298 INFO:teuthology.orchestra.run.smithi195.stdout:Nothing to do. 2021-03-09T08:47:13.299 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:13.381 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-fuse 2021-03-09T08:47:13.926 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:06 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:14.761 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:14.762 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:14.762 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2021-03-09T08:47:14.763 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:14.763 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:47:14.763 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 86 k 2021-03-09T08:47:14.763 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:14.763 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:47:14.764 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:14.764 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2021-03-09T08:47:14.764 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:14.764 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 86 k 2021-03-09T08:47:14.765 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 298 k 2021-03-09T08:47:14.765 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:47:14.908 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-fuse-17.0.0-1742.g89fb622b.el8.x86_64.rpm 590 kB/s | 86 kB 00:00 2021-03-09T08:47:14.910 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:47:14.910 INFO:teuthology.orchestra.run.smithi195.stdout:Total 585 kB/s | 86 kB 00:00 2021-03-09T08:47:14.911 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:47:14.921 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:47:14.922 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:47:14.971 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:47:14.974 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:47:15.079 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:47:15.293 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:15.527 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:15.771 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:15.771 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:47:16.020 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:16.021 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:47:16.021 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-fuse-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:47:16.021 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:16.021 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:16.271 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-mirror 2021-03-09T08:47:16.819 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:09 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:17.511 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:17.512 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:17.512 INFO:teuthology.orchestra.run.smithi195.stdout: Package Arch Version Repo Size 2021-03-09T08:47:17.512 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:17.512 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:47:17.513 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 3.4 M 2021-03-09T08:47:17.513 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:17.513 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:47:17.513 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:17.514 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2021-03-09T08:47:17.514 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:17.515 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.4 M 2021-03-09T08:47:17.515 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 18 M 2021-03-09T08:47:17.515 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:47:17.985 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-mirror-17.0.0-1742.g89fb622b.el8.x86_64.rpm 7.1 MB/s | 3.4 MB 00:00 2021-03-09T08:47:17.986 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:47:17.986 INFO:teuthology.orchestra.run.smithi195.stdout:Total 7.1 MB/s | 3.4 MB 00:00 2021-03-09T08:47:17.987 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:47:18.000 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:47:18.000 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:47:18.064 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:47:18.067 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:47:18.515 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:47:18.746 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-mirror-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:19.107 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-mirror-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:19.358 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-mirror-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:19.358 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:47:19.775 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:19.776 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:47:19.776 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-mirror-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:47:19.776 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:19.776 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:20.107 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum -y install rbd-nbd 2021-03-09T08:47:20.654 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:02:13 ago on Tue 09 Mar 2021 08:45:07 AM UTC. 2021-03-09T08:47:21.356 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2021-03-09T08:47:21.357 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:21.357 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2021-03-09T08:47:21.357 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:21.358 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2021-03-09T08:47:21.358 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd x86_64 2:17.0.0-1742.g89fb622b.el8 ceph 165 k 2021-03-09T08:47:21.358 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:21.358 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2021-03-09T08:47:21.358 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2021-03-09T08:47:21.359 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2021-03-09T08:47:21.359 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:21.359 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 165 k 2021-03-09T08:47:21.360 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 649 k 2021-03-09T08:47:21.360 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2021-03-09T08:47:21.526 INFO:teuthology.orchestra.run.smithi195.stdout:rbd-nbd-17.0.0-1742.g89fb622b.el8.x86_64.rpm 979 kB/s | 165 kB 00:00 2021-03-09T08:47:21.528 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2021-03-09T08:47:21.528 INFO:teuthology.orchestra.run.smithi195.stdout:Total 971 kB/s | 165 kB 00:00 2021-03-09T08:47:21.528 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2021-03-09T08:47:21.540 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2021-03-09T08:47:21.541 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2021-03-09T08:47:21.592 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2021-03-09T08:47:21.595 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2021-03-09T08:47:21.722 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2021-03-09T08:47:21.934 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : rbd-nbd-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:22.194 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: rbd-nbd-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:22.470 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : rbd-nbd-2:17.0.0-1742.g89fb622b.el8.x86_64 1/1 2021-03-09T08:47:22.470 INFO:teuthology.orchestra.run.smithi195.stdout:Installed products updated. 2021-03-09T08:47:22.715 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:22.715 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2021-03-09T08:47:22.715 INFO:teuthology.orchestra.run.smithi195.stdout: rbd-nbd-2:17.0.0-1742.g89fb622b.el8.x86_64 2021-03-09T08:47:22.715 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:22.716 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2021-03-09T08:47:22.878 DEBUG:teuthology.parallel:result is None 2021-03-09T08:47:22.879 DEBUG:teuthology.orchestra.run.smithi099:> sudo lsb_release -is 2021-03-09T08:47:22.921 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS 2021-03-09T08:47:22.922 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-09T08:47:22.923 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:47:23.073 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-03-09T08:47:23.097 INFO:teuthology.orchestra.run.smithi099.stdout:17.0.0-1742.g89fb622b.el8 2021-03-09T08:47:23.097 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742.g89fb622b.el8 2021-03-09T08:47:23.098 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742.g89fb622b is installed. 2021-03-09T08:47:23.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo lsb_release -is 2021-03-09T08:47:23.132 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS 2021-03-09T08:47:23.133 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-09T08:47:23.134 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=89fb622b10ca2e58fe0c913a8956377a886e2ab4 2021-03-09T08:47:23.281 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-03-09T08:47:23.300 INFO:teuthology.orchestra.run.smithi195.stdout:17.0.0-1742.g89fb622b.el8 2021-03-09T08:47:23.301 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-1742.g89fb622b.el8 2021-03-09T08:47:23.301 INFO:teuthology.task.install:The correct ceph version 17.0.0-1742.g89fb622b is installed. 2021-03-09T08:47:23.302 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-09T08:47:23.306 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:23.306 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T08:47:23.335 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:23.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-09T08:47:23.364 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-09T08:47:23.366 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:23.366 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T08:47:23.402 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T08:47:23.468 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:23.468 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2021-03-09T08:47:23.493 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-09T08:47:23.556 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-09T08:47:23.557 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:23.557 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T08:47:23.583 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T08:47:23.648 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:23.648 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-09T08:47:23.672 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-09T08:47:23.736 INFO:teuthology.run_tasks:Running task ceph... 2021-03-09T08:47:23.817 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-03-09T08:47:23.818 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /var/log/ceph 2021-03-09T08:47:23.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /var/log/ceph 2021-03-09T08:47:23.845 INFO:tasks.ceph:Disabling ceph logrotate... 2021-03-09T08:47:23.845 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-09T08:47:23.885 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-09T08:47:23.909 INFO:tasks.ceph:Creating extra log directories... 2021-03-09T08:47:23.909 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-09T08:47:23.950 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-09T08:47:24.117 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-03-09T08:47:24.118 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug monc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-03-09T08:47:24.118 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi/5949083', 'branch': 'wip-kefu-testing-2021-03-09-1428', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados tasks/rados_api_tests validater/lockdep}', 'email': 'tchaikov@gmail.com', 'first_in_suite': False, 'job_id': '5949083', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'kchai-2021-03-09_08:22:40-rados-wip-kefu-testing-2021-03-09-1428-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.3', 'overrides': {'admin_socket': {'branch': 'wip-kefu-testing-2021-03-09-1428'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug monc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-kefu-testing-2021-03-09-1428', 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4'}}, 'owner': 'scheduled_kchai@teuthology', 'priority': 60, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-kefu-testing-2021-03-09-1428', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_89fb622b10ca2e58fe0c913a8956377a886e2ab4/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '89fb622b10ca2e58fe0c913a8956377a886e2ab4', 'targets': {'smithi099.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDg1b1HraNq0AWyoPGdbrFvsAebk5Cfx+u2LSwiCX81xITsIQImbZN3B1l7yxhIaVH7fSKirZjbIYmwJyr8NxjRex05vaa59nNTD4LNJBh+Q71nKGnf/AEp8MlXkMwdlQWSWIIodcuSI+TD6pVi4vnYfpqeJBa11cisUnPeHLk8I9FiI4byUVR561j/Zvbx1h0QlzbthtnL8Hai3NLWRMpwjVmVdAeRPaeMJGIz5oo4ho8YDo5y2I16u0NwVLTsd+OYasmj/tyx3oyy7BV8V8Aw1opyWEG8eZ6lhhXKVtw/Q1IulfN/NMh+MgUkaDm2hmJdriHj5k4cpBurxXEqXbBrpgvvoiCEOVGQwzAM//dR6jZOwRkMIiWt3IZ9O1bq/NSs86HPEK/iEvqTntw0UpcNuybBLRT+JgN3aizF6i8HsBMPOT8tVz9qFea+1koIOgvWoy9gtAe5N5A1xDupjl00r79Nt01kczSMxCRYJnMU2W39MIQRedsgjGOj0sLWU5k=', 'smithi195.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6vWtGsGfQ4Z1POo2m1ZI+yBkT5G9TWJiO6WZt4CYAtyc9y2Mt3Nr4jp7dRiQNJke0xAGppqPvol+5C9vn2Tdm2vTrpzgustdXvwwJaOxxQnPL4oJEJLE3e2EUjTRFHJVdcMc/NUQm08FOLgV4HZTP6wM3n7nl4n1VEZaM4Dl3Ndd78TH9288qVZiEbRYPqJzHj+iYKQEgZN6RluEthreT7XYBP/djZVIDqXS4uxVVLL/IeyT100Rzq8/BlE89zMJbH0sW5PeS78pygHQQN/bICycJEDzkWwnKVCqIiQYruD5+E8Af1pcrfxWuXElsfN4IMe234Ia/vtzuazB5WEWACsNkuPIYu81pSvqNpFlIeYXGWcB/dlIHZwYPW+iybTpYQcvkRnutkBvNl296MToqwIcap/pMbhyL4fSJnCa87tZovjwAJQOZk3kkUzOobqbXISqKlAOTretXIdT//dcqvDcqoPLfEnNxJoaiukgV4plHDVMvAK1LVzKBjW+x9+M='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology_branch': 'master', 'teuthology_sha1': '3082387bba74fcd24c9700593d10418152d53c97', 'timestamp': '2021-03-09_08:22:40', 'tube': 'smithi', 'user': 'kchai', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.25973'} 2021-03-09T08:47:24.119 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-03-09T08:47:24.122 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-03-09T08:47:24.137 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m0777 -- /var/run/ceph 2021-03-09T08:47:24.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2021-03-09T08:47:24.202 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:24.202 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T08:47:24.254 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T08:47:24.255 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T08:47:24.309 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T08:47:24.309 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:47:24.309 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 98447 Links: 1 2021-03-09T08:47:24.310 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:47:24.310 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2021-03-09T08:47:24.310 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2021-03-09 08:46:43.281879698 +0000 2021-03-09T08:47:24.310 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2021-03-09 08:42:58.100670480 +0000 2021-03-09T08:47:24.311 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2021-03-09 08:42:58.100670480 +0000 2021-03-09T08:47:24.311 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2021-03-09T08:47:24.318 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2021-03-09T08:47:24.319 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2021-03-09T08:47:24.319 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000246851 s, 2.1 MB/s 2021-03-09T08:47:24.321 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T08:47:24.376 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T08:47:24.376 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:47:24.377 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 95087 Links: 1 2021-03-09T08:47:24.377 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:47:24.377 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2021-03-09T08:47:24.377 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2021-03-09 08:46:43.281879698 +0000 2021-03-09T08:47:24.378 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2021-03-09 08:42:58.439668890 +0000 2021-03-09T08:47:24.378 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2021-03-09 08:42:58.439668890 +0000 2021-03-09T08:47:24.378 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2021-03-09T08:47:24.386 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2021-03-09T08:47:24.386 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2021-03-09T08:47:24.386 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.00023802 s, 2.2 MB/s 2021-03-09T08:47:24.388 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T08:47:24.442 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T08:47:24.442 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:47:24.443 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 95214 Links: 1 2021-03-09T08:47:24.443 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:47:24.443 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2021-03-09T08:47:24.443 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2021-03-09 08:46:43.281879698 +0000 2021-03-09T08:47:24.444 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2021-03-09 08:42:58.757667398 +0000 2021-03-09T08:47:24.444 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2021-03-09 08:42:58.757667398 +0000 2021-03-09T08:47:24.444 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2021-03-09T08:47:24.452 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2021-03-09T08:47:24.452 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2021-03-09T08:47:24.452 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000244603 s, 2.1 MB/s 2021-03-09T08:47:24.454 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T08:47:24.510 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T08:47:24.510 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:47:24.510 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 6h/6d Inode: 98577 Links: 1 2021-03-09T08:47:24.511 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:47:24.511 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2021-03-09T08:47:24.511 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2021-03-09 08:46:43.282879693 +0000 2021-03-09T08:47:24.512 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2021-03-09 08:42:59.079665888 +0000 2021-03-09T08:47:24.512 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2021-03-09 08:42:59.079665888 +0000 2021-03-09T08:47:24.512 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: - 2021-03-09T08:47:24.519 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2021-03-09T08:47:24.519 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2021-03-09T08:47:24.520 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000284928 s, 1.8 MB/s 2021-03-09T08:47:24.523 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:24.523 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:24.524 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2021-03-09T08:47:24.540 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-09T08:47:24.540 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-09T08:47:24.595 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-09T08:47:24.595 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:47:24.596 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 94994 Links: 1 2021-03-09T08:47:24.596 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:47:24.596 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2021-03-09T08:47:24.596 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2021-03-09 08:47:18.913940368 +0000 2021-03-09T08:47:24.597 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2021-03-09 08:42:58.378061554 +0000 2021-03-09T08:47:24.597 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2021-03-09 08:42:58.378061554 +0000 2021-03-09T08:47:24.597 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2021-03-09T08:47:24.605 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2021-03-09T08:47:24.605 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2021-03-09T08:47:24.605 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000130652 s, 3.9 MB/s 2021-03-09T08:47:24.608 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-09T08:47:24.663 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-09T08:47:24.663 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:47:24.663 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 97625 Links: 1 2021-03-09T08:47:24.664 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:47:24.664 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2021-03-09T08:47:24.664 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2021-03-09 08:47:18.913940368 +0000 2021-03-09T08:47:24.666 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2021-03-09 08:42:58.727054996 +0000 2021-03-09T08:47:24.666 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2021-03-09 08:42:58.727054996 +0000 2021-03-09T08:47:24.666 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2021-03-09T08:47:24.673 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2021-03-09T08:47:24.673 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2021-03-09T08:47:24.674 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000136381 s, 3.8 MB/s 2021-03-09T08:47:24.676 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-09T08:47:24.730 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-09T08:47:24.730 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:47:24.730 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 95891 Links: 1 2021-03-09T08:47:24.731 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:47:24.731 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2021-03-09T08:47:24.731 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2021-03-09 08:47:18.913940368 +0000 2021-03-09T08:47:24.731 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2021-03-09 08:42:59.059048758 +0000 2021-03-09T08:47:24.732 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2021-03-09 08:42:59.059048758 +0000 2021-03-09T08:47:24.732 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2021-03-09T08:47:24.739 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2021-03-09T08:47:24.739 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2021-03-09T08:47:24.740 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000110556 s, 4.6 MB/s 2021-03-09T08:47:24.742 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-09T08:47:24.797 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-09T08:47:24.797 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-09T08:47:24.798 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 6h/6d Inode: 98735 Links: 1 2021-03-09T08:47:24.798 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-09T08:47:24.798 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2021-03-09T08:47:24.799 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2021-03-09 08:47:18.913940368 +0000 2021-03-09T08:47:24.799 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2021-03-09 08:42:59.369042933 +0000 2021-03-09T08:47:24.799 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2021-03-09 08:42:59.369042933 +0000 2021-03-09T08:47:24.799 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2021-03-09T08:47:24.806 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2021-03-09T08:47:24.807 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2021-03-09T08:47:24.807 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000101562 s, 5.0 MB/s 2021-03-09T08:47:24.810 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:24.810 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-03-09T08:47:24.811 INFO:tasks.ceph:Generating config... 2021-03-09T08:47:24.814 INFO:tasks.ceph:[client] debug monc = 20 2021-03-09T08:47:24.815 INFO:tasks.ceph:[client] debug ms = 1 2021-03-09T08:47:24.815 INFO:tasks.ceph:[client] debug objecter = 20 2021-03-09T08:47:24.816 INFO:tasks.ceph:[client] debug rados = 20 2021-03-09T08:47:24.816 INFO:tasks.ceph:[global] lockdep = True 2021-03-09T08:47:24.817 INFO:tasks.ceph:[global] mon client directed command retry = 5 2021-03-09T08:47:24.818 INFO:tasks.ceph:[global] mon election default strategy = 3 2021-03-09T08:47:24.818 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2021-03-09T08:47:24.819 INFO:tasks.ceph:[global] ms type = async 2021-03-09T08:47:24.819 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-09T08:47:24.820 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-09T08:47:24.820 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-09T08:47:24.821 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-09T08:47:24.821 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-09T08:47:24.822 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2021-03-09T08:47:24.822 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2021-03-09T08:47:24.823 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2021-03-09T08:47:24.823 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2021-03-09T08:47:24.824 INFO:tasks.ceph:[mon] mon scrub interval = 300 2021-03-09T08:47:24.824 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2021-03-09T08:47:24.824 INFO:tasks.ceph:[mon] paxos service trim min = 10 2021-03-09T08:47:24.825 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-03-09T08:47:24.825 INFO:tasks.ceph:[osd] bluestore compression algorithm = lz4 2021-03-09T08:47:24.826 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2021-03-09T08:47:24.826 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-03-09T08:47:24.827 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-03-09T08:47:24.827 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-03-09T08:47:24.827 INFO:tasks.ceph:[osd] debug filestore = 20 2021-03-09T08:47:24.827 INFO:tasks.ceph:[osd] debug journal = 20 2021-03-09T08:47:24.828 INFO:tasks.ceph:[osd] debug monc = 20 2021-03-09T08:47:24.828 INFO:tasks.ceph:[osd] debug ms = 20 2021-03-09T08:47:24.828 INFO:tasks.ceph:[osd] debug osd = 25 2021-03-09T08:47:24.829 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-03-09T08:47:24.829 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-03-09T08:47:24.829 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-03-09T08:47:24.829 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-03-09T08:47:24.830 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2021-03-09T08:47:24.830 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2021-03-09T08:47:24.830 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-03-09T08:47:24.831 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2021-03-09T08:47:24.831 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-03-09T08:47:24.831 INFO:tasks.ceph:[osd] osd op queue = debug_random 2021-03-09T08:47:24.831 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2021-03-09T08:47:24.832 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2021-03-09T08:47:24.832 INFO:tasks.ceph:Setting up mon.a... 2021-03-09T08:47:24.833 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-03-09T08:47:24.937 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.keyring 2021-03-09T08:47:24.940 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-03-09T08:47:25.001 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T08:47:25.025 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.99'), ('mon.c', '[v2:172.21.15.99:3301,v1:172.21.15.99:6790]'), ('mon.b', '172.21.15.195')] 2021-03-09T08:47:25.026 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.99,[v2:172.21.15.99:3301,v1:172.21.15.99:6790],172.21.15.195', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'lz4', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug monc': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2021-03-09T08:47:25.027 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:25.028 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-03-09T08:47:25.079 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.99 --addv c '[v2:172.21.15.99:3301,v1:172.21.15.99:6790]' --add b 172.21.15.195 --print /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:25.177 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:25.177 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: generated fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:47:25.178 INFO:teuthology.orchestra.run.smithi099.stdout:setting min_mon_release = octopus 2021-03-09T08:47:25.178 INFO:teuthology.orchestra.run.smithi099.stdout:epoch 0 2021-03-09T08:47:25.179 INFO:teuthology.orchestra.run.smithi099.stdout:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:47:25.179 INFO:teuthology.orchestra.run.smithi099.stdout:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:47:25.179 INFO:teuthology.orchestra.run.smithi099.stdout:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:47:25.180 INFO:teuthology.orchestra.run.smithi099.stdout:min_mon_release 15 (octopus) 2021-03-09T08:47:25.180 INFO:teuthology.orchestra.run.smithi099.stdout:election_strategy: 3 2021-03-09T08:47:25.181 INFO:teuthology.orchestra.run.smithi099.stdout:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:47:25.181 INFO:teuthology.orchestra.run.smithi099.stdout:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:47:25.182 INFO:teuthology.orchestra.run.smithi099.stdout:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:47:25.182 INFO:teuthology.orchestra.run.smithi099.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-03-09T08:47:25.185 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-03-09T08:47:25.199 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e7417d55-b73a-46ac-8f0e-ad6e583f612d... 2021-03-09T08:47:25.201 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-03-09T08:47:25.242 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-03-09T08:47:25.281 INFO:teuthology.orchestra.run.smithi099.stdout:[global] 2021-03-09T08:47:25.281 INFO:teuthology.orchestra.run.smithi099.stdout: chdir = "" 2021-03-09T08:47:25.282 INFO:teuthology.orchestra.run.smithi099.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-09T08:47:25.282 INFO:teuthology.orchestra.run.smithi099.stdout: auth supported = cephx 2021-03-09T08:47:25.282 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.282 INFO:teuthology.orchestra.run.smithi099.stdout: filestore xattr use omap = true 2021-03-09T08:47:25.283 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.283 INFO:teuthology.orchestra.run.smithi099.stdout: mon clock drift allowed = 1.000 2021-03-09T08:47:25.283 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.283 INFO:teuthology.orchestra.run.smithi099.stdout: osd crush chooseleaf type = 0 2021-03-09T08:47:25.284 INFO:teuthology.orchestra.run.smithi099.stdout: auth debug = true 2021-03-09T08:47:25.284 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.284 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on old message = true 2021-03-09T08:47:25.284 INFO:teuthology.orchestra.run.smithi099.stdout: ms die on bug = true 2021-03-09T08:47:25.284 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.285 INFO:teuthology.orchestra.run.smithi099.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-09T08:47:25.285 INFO:teuthology.orchestra.run.smithi099.stdout: mon pg warn max object skew = 0 2021-03-09T08:47:25.285 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.286 INFO:teuthology.orchestra.run.smithi099.stdout: # disable pg_autoscaler by default for new pools 2021-03-09T08:47:25.286 INFO:teuthology.orchestra.run.smithi099.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-09T08:47:25.286 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.286 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default size = 2 2021-03-09T08:47:25.286 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.287 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow primary affinity = true 2021-03-09T08:47:25.287 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd allow pg remap = true 2021-03-09T08:47:25.287 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on legacy crush tunables = false 2021-03-09T08:47:25.287 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on crush straw calc version zero = false 2021-03-09T08:47:25.288 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on no sortbitwise = false 2021-03-09T08:47:25.288 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on osd down out interval zero = false 2021-03-09T08:47:25.288 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on too few osds = false 2021-03-09T08:47:25.288 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-09T08:47:25.288 INFO:teuthology.orchestra.run.smithi099.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-09T08:47:25.289 INFO:teuthology.orchestra.run.smithi099.stdout: mon_allow_pool_size_one = true 2021-03-09T08:47:25.289 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.289 INFO:teuthology.orchestra.run.smithi099.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-09T08:47:25.289 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.289 INFO:teuthology.orchestra.run.smithi099.stdout: osd default data pool replay window = 5 2021-03-09T08:47:25.290 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.290 INFO:teuthology.orchestra.run.smithi099.stdout: mon allow pool delete = true 2021-03-09T08:47:25.290 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: mon cluster log file level = debug 2021-03-09T08:47:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: debug asserts on shutdown = true 2021-03-09T08:47:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: mon health detail to clog = false 2021-03-09T08:47:25.291 INFO:teuthology.orchestra.run.smithi099.stdout: mon host = "172.21.15.99,[v2:172.21.15.99:3301,v1:172.21.15.99:6790],172.21.15.195" 2021-03-09T08:47:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: lockdep = True 2021-03-09T08:47:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: mon client directed command retry = 5 2021-03-09T08:47:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: mon election default strategy = 3 2021-03-09T08:47:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: ms inject socket failures = 5000 2021-03-09T08:47:25.292 INFO:teuthology.orchestra.run.smithi099.stdout: ms type = async 2021-03-09T08:47:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: fsid = e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:47:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.293 INFO:teuthology.orchestra.run.smithi099.stdout:[osd] 2021-03-09T08:47:25.293 INFO:teuthology.orchestra.run.smithi099.stdout: osd journal size = 100 2021-03-09T08:47:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub load threshold = 5.0 2021-03-09T08:47:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: osd scrub max interval = 600 2021-03-09T08:47:25.294 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: osd recover clone overlap = true 2021-03-09T08:47:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: osd recovery max chunk = 1048576 2021-03-09T08:47:25.295 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.296 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug shutdown = true 2021-03-09T08:47:25.296 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug op order = true 2021-03-09T08:47:25.296 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify stray on activate = true 2021-03-09T08:47:25.297 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.297 INFO:teuthology.orchestra.run.smithi099.stdout: osd open classes on start = true 2021-03-09T08:47:25.297 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug pg log writeout = true 2021-03-09T08:47:25.297 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.297 INFO:teuthology.orchestra.run.smithi099.stdout: osd deep scrub update digest min age = 30 2021-03-09T08:47:25.298 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.298 INFO:teuthology.orchestra.run.smithi099.stdout: osd map max advance = 10 2021-03-09T08:47:25.298 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.298 INFO:teuthology.orchestra.run.smithi099.stdout: journal zero on create = true 2021-03-09T08:47:25.298 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.299 INFO:teuthology.orchestra.run.smithi099.stdout: filestore ondisk finisher threads = 3 2021-03-09T08:47:25.299 INFO:teuthology.orchestra.run.smithi099.stdout: filestore apply finisher threads = 3 2021-03-09T08:47:25.299 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.299 INFO:teuthology.orchestra.run.smithi099.stdout: bdev debug aio = true 2021-03-09T08:47:25.300 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug misdirected ops = true 2021-03-09T08:47:25.300 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore block size = 96636764160 2021-03-09T08:47:25.300 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore compression algorithm = lz4 2021-03-09T08:47:25.300 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore compression mode = aggressive 2021-03-09T08:47:25.300 INFO:teuthology.orchestra.run.smithi099.stdout: bluestore fsck on mount = True 2021-03-09T08:47:25.301 INFO:teuthology.orchestra.run.smithi099.stdout: debug bluefs = 20 2021-03-09T08:47:25.301 INFO:teuthology.orchestra.run.smithi099.stdout: debug bluestore = 20 2021-03-09T08:47:25.301 INFO:teuthology.orchestra.run.smithi099.stdout: debug filestore = 20 2021-03-09T08:47:25.301 INFO:teuthology.orchestra.run.smithi099.stdout: debug journal = 20 2021-03-09T08:47:25.302 INFO:teuthology.orchestra.run.smithi099.stdout: debug monc = 20 2021-03-09T08:47:25.302 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 20 2021-03-09T08:47:25.302 INFO:teuthology.orchestra.run.smithi099.stdout: debug osd = 25 2021-03-09T08:47:25.303 INFO:teuthology.orchestra.run.smithi099.stdout: debug rocksdb = 10 2021-03-09T08:47:25.303 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd backfillfull_ratio = 0.85 2021-03-09T08:47:25.303 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd full ratio = 0.9 2021-03-09T08:47:25.303 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd nearfull ratio = 0.8 2021-03-09T08:47:25.303 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify cached snaps = True 2021-03-09T08:47:25.304 INFO:teuthology.orchestra.run.smithi099.stdout: osd debug verify missing on start = True 2021-03-09T08:47:25.304 INFO:teuthology.orchestra.run.smithi099.stdout: osd failsafe full ratio = 0.95 2021-03-09T08:47:25.304 INFO:teuthology.orchestra.run.smithi099.stdout: osd max markdown count = 1000 2021-03-09T08:47:25.304 INFO:teuthology.orchestra.run.smithi099.stdout: osd objectstore = bluestore 2021-03-09T08:47:25.304 INFO:teuthology.orchestra.run.smithi099.stdout: osd op queue = debug_random 2021-03-09T08:47:25.305 INFO:teuthology.orchestra.run.smithi099.stdout: osd op queue cut off = debug_random 2021-03-09T08:47:25.305 INFO:teuthology.orchestra.run.smithi099.stdout: osd shutdown pgref assert = True 2021-03-09T08:47:25.305 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.306 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr] 2021-03-09T08:47:25.306 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2021-03-09T08:47:25.306 INFO:teuthology.orchestra.run.smithi099.stdout: debug mgr = 20 2021-03-09T08:47:25.306 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2021-03-09T08:47:25.306 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2021-03-09T08:47:25.307 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2021-03-09T08:47:25.307 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2021-03-09T08:47:25.307 INFO:teuthology.orchestra.run.smithi099.stdout: mgr/telemetry/nag = false 2021-03-09T08:47:25.307 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.307 INFO:teuthology.orchestra.run.smithi099.stdout:[mon] 2021-03-09T08:47:25.308 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2021-03-09T08:47:25.308 INFO:teuthology.orchestra.run.smithi099.stdout: debug mon = 20 2021-03-09T08:47:25.308 INFO:teuthology.orchestra.run.smithi099.stdout: debug paxos = 20 2021-03-09T08:47:25.308 INFO:teuthology.orchestra.run.smithi099.stdout: debug auth = 20 2021-03-09T08:47:25.309 INFO:teuthology.orchestra.run.smithi099.stdout: mon data avail warn = 5 2021-03-09T08:47:25.309 INFO:teuthology.orchestra.run.smithi099.stdout: mon mgr mkfs grace = 240 2021-03-09T08:47:25.309 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min pgs per osd = 4 2021-03-09T08:47:25.309 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd reporter subtree level = osd 2021-03-09T08:47:25.310 INFO:teuthology.orchestra.run.smithi099.stdout: mon osd prime pg temp = true 2021-03-09T08:47:25.310 INFO:teuthology.orchestra.run.smithi099.stdout: mon reweight min bytes per osd = 10 2021-03-09T08:47:25.310 INFO:teuthology.orchestra.run.smithi099.stdout: mon min osdmap epochs = 50 2021-03-09T08:47:25.310 INFO:teuthology.orchestra.run.smithi099.stdout: mon osdmap full prune interval = 2 2021-03-09T08:47:25.311 INFO:teuthology.orchestra.run.smithi099.stdout: mon osdmap full prune min = 15 2021-03-09T08:47:25.311 INFO:teuthology.orchestra.run.smithi099.stdout: mon osdmap full prune txsize = 2 2021-03-09T08:47:25.311 INFO:teuthology.orchestra.run.smithi099.stdout: mon scrub interval = 300 2021-03-09T08:47:25.311 INFO:teuthology.orchestra.run.smithi099.stdout: mon warn on pool no app = False 2021-03-09T08:47:25.312 INFO:teuthology.orchestra.run.smithi099.stdout: paxos service trim min = 10 2021-03-09T08:47:25.312 INFO:teuthology.orchestra.run.smithi099.stdout: 2021-03-09T08:47:25.312 INFO:teuthology.orchestra.run.smithi099.stdout:[client] 2021-03-09T08:47:25.312 INFO:teuthology.orchestra.run.smithi099.stdout: rgw cache enabled = true 2021-03-09T08:47:25.312 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable ops log = true 2021-03-09T08:47:25.313 INFO:teuthology.orchestra.run.smithi099.stdout: rgw enable usage log = true 2021-03-09T08:47:25.313 INFO:teuthology.orchestra.run.smithi099.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-09T08:47:25.313 INFO:teuthology.orchestra.run.smithi099.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-09T08:47:25.313 INFO:teuthology.orchestra.run.smithi099.stdout: debug monc = 20 2021-03-09T08:47:25.313 INFO:teuthology.orchestra.run.smithi099.stdout: debug ms = 1 2021-03-09T08:47:25.314 INFO:teuthology.orchestra.run.smithi099.stdout: debug objecter = 20 2021-03-09T08:47:25.314 INFO:teuthology.orchestra.run.smithi099.stdout: debug rados = 20 2021-03-09T08:47:25.314 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.a] 2021-03-09T08:47:25.314 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.c] 2021-03-09T08:47:25.315 INFO:teuthology.orchestra.run.smithi099.stdout:[mon.b] 2021-03-09T08:47:25.317 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2021-03-09T08:47:25.317 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2021-03-09T08:47:25.317 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-09T08:47:25.317 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2021-03-09T08:47:25.318 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.318 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2021-03-09T08:47:25.318 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.318 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2021-03-09T08:47:25.318 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.319 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2021-03-09T08:47:25.319 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2021-03-09T08:47:25.319 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.320 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2021-03-09T08:47:25.320 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2021-03-09T08:47:25.320 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.320 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-09T08:47:25.320 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2021-03-09T08:47:25.321 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.321 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2021-03-09T08:47:25.321 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-09T08:47:25.321 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.321 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2021-03-09T08:47:25.322 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.322 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2021-03-09T08:47:25.322 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2021-03-09T08:47:25.323 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2021-03-09T08:47:25.323 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2021-03-09T08:47:25.323 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2021-03-09T08:47:25.323 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2021-03-09T08:47:25.323 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2021-03-09T08:47:25.324 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-09T08:47:25.324 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-09T08:47:25.324 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2021-03-09T08:47:25.324 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.325 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-09T08:47:25.325 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.325 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2021-03-09T08:47:25.325 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.326 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2021-03-09T08:47:25.326 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.326 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2021-03-09T08:47:25.326 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2021-03-09T08:47:25.326 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2021-03-09T08:47:25.327 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = "172.21.15.99,[v2:172.21.15.99:3301,v1:172.21.15.99:6790],172.21.15.195" 2021-03-09T08:47:25.327 INFO:teuthology.orchestra.run.smithi195.stdout: lockdep = True 2021-03-09T08:47:25.327 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2021-03-09T08:47:25.327 INFO:teuthology.orchestra.run.smithi195.stdout: mon election default strategy = 3 2021-03-09T08:47:25.327 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 5000 2021-03-09T08:47:25.328 INFO:teuthology.orchestra.run.smithi195.stdout: ms type = async 2021-03-09T08:47:25.328 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:47:25.328 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.328 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2021-03-09T08:47:25.329 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2021-03-09T08:47:25.329 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.329 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2021-03-09T08:47:25.329 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 600 2021-03-09T08:47:25.330 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.330 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2021-03-09T08:47:25.330 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2021-03-09T08:47:25.330 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.330 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2021-03-09T08:47:25.331 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2021-03-09T08:47:25.331 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2021-03-09T08:47:25.331 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.331 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2021-03-09T08:47:25.332 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2021-03-09T08:47:25.332 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.332 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2021-03-09T08:47:25.332 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.333 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2021-03-09T08:47:25.333 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.333 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2021-03-09T08:47:25.333 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.333 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2021-03-09T08:47:25.334 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2021-03-09T08:47:25.334 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.334 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2021-03-09T08:47:25.335 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2021-03-09T08:47:25.335 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2021-03-09T08:47:25.335 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore compression algorithm = lz4 2021-03-09T08:47:25.335 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore compression mode = aggressive 2021-03-09T08:47:25.335 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2021-03-09T08:47:25.336 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 20 2021-03-09T08:47:25.336 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 20 2021-03-09T08:47:25.336 INFO:teuthology.orchestra.run.smithi195.stdout: debug filestore = 20 2021-03-09T08:47:25.336 INFO:teuthology.orchestra.run.smithi195.stdout: debug journal = 20 2021-03-09T08:47:25.336 INFO:teuthology.orchestra.run.smithi195.stdout: debug monc = 20 2021-03-09T08:47:25.337 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 20 2021-03-09T08:47:25.337 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 25 2021-03-09T08:47:25.337 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 10 2021-03-09T08:47:25.337 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2021-03-09T08:47:25.337 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2021-03-09T08:47:25.338 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2021-03-09T08:47:25.338 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify cached snaps = True 2021-03-09T08:47:25.338 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify missing on start = True 2021-03-09T08:47:25.339 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2021-03-09T08:47:25.339 INFO:teuthology.orchestra.run.smithi195.stdout: osd max markdown count = 1000 2021-03-09T08:47:25.339 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2021-03-09T08:47:25.339 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue = debug_random 2021-03-09T08:47:25.340 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue cut off = debug_random 2021-03-09T08:47:25.340 INFO:teuthology.orchestra.run.smithi195.stdout: osd shutdown pgref assert = True 2021-03-09T08:47:25.340 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.340 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2021-03-09T08:47:25.341 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2021-03-09T08:47:25.341 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2021-03-09T08:47:25.341 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2021-03-09T08:47:25.341 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2021-03-09T08:47:25.341 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2021-03-09T08:47:25.342 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2021-03-09T08:47:25.342 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2021-03-09T08:47:25.342 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.342 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2021-03-09T08:47:25.342 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2021-03-09T08:47:25.343 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2021-03-09T08:47:25.343 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2021-03-09T08:47:25.343 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2021-03-09T08:47:25.344 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2021-03-09T08:47:25.344 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2021-03-09T08:47:25.344 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2021-03-09T08:47:25.344 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2021-03-09T08:47:25.344 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2021-03-09T08:47:25.345 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2021-03-09T08:47:25.345 INFO:teuthology.orchestra.run.smithi195.stdout: mon min osdmap epochs = 50 2021-03-09T08:47:25.345 INFO:teuthology.orchestra.run.smithi195.stdout: mon osdmap full prune interval = 2 2021-03-09T08:47:25.345 INFO:teuthology.orchestra.run.smithi195.stdout: mon osdmap full prune min = 15 2021-03-09T08:47:25.345 INFO:teuthology.orchestra.run.smithi195.stdout: mon osdmap full prune txsize = 2 2021-03-09T08:47:25.346 INFO:teuthology.orchestra.run.smithi195.stdout: mon scrub interval = 300 2021-03-09T08:47:25.346 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on pool no app = False 2021-03-09T08:47:25.346 INFO:teuthology.orchestra.run.smithi195.stdout: paxos service trim min = 10 2021-03-09T08:47:25.346 INFO:teuthology.orchestra.run.smithi195.stdout: 2021-03-09T08:47:25.347 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2021-03-09T08:47:25.347 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2021-03-09T08:47:25.347 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2021-03-09T08:47:25.347 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2021-03-09T08:47:25.348 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-09T08:47:25.348 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-09T08:47:25.348 INFO:teuthology.orchestra.run.smithi195.stdout: debug monc = 20 2021-03-09T08:47:25.348 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2021-03-09T08:47:25.348 INFO:teuthology.orchestra.run.smithi195.stdout: debug objecter = 20 2021-03-09T08:47:25.349 INFO:teuthology.orchestra.run.smithi195.stdout: debug rados = 20 2021-03-09T08:47:25.349 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2021-03-09T08:47:25.349 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.c] 2021-03-09T08:47:25.349 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.b] 2021-03-09T08:47:25.350 INFO:tasks.ceph:Creating admin key on mon.a... 2021-03-09T08:47:25.350 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-03-09T08:47:25.424 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-09T08:47:25.425 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:25.426 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2021-03-09T08:47:25.443 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:25.444 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2021-03-09T08:47:25.497 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi099.front.sepia.ceph.com 2021-03-09T08:47:25.497 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:25.498 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.keyring 2021-03-09T08:47:25.498 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T08:47:25.570 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:25.570 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:25.624 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2021-03-09T08:47:25.624 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:25.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.keyring 2021-03-09T08:47:25.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-03-09T08:47:25.660 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:25.661 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:25.716 INFO:tasks.ceph:Setting up mon nodes... 2021-03-09T08:47:25.716 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-09T08:47:25.717 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2021-03-09T08:47:25.781 INFO:teuthology.orchestra.run.smithi099.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2021-03-09T08:47:25.784 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-03-09T08:47:25.904 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-03-09T08:47:25.907 INFO:tasks.ceph:Setting up mds nodes... 2021-03-09T08:47:25.908 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-09T08:47:25.909 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-03-09T08:47:25.964 INFO:teuthology.orchestra.run.smithi099.stdout:creating /etc/ceph/ceph.client.0.keyring 2021-03-09T08:47:25.978 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-03-09T08:47:26.033 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/ceph.client.1.keyring 2021-03-09T08:47:26.046 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-09T08:47:26.047 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi099.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2021-03-09T08:47:26.047 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-03-09T08:47:26.071 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:26.072 INFO:tasks.ceph:role: osd.0 2021-03-09T08:47:26.072 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com 2021-03-09T08:47:26.073 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-09T08:47:26.142 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:47:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:47:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-03-09T08:47:26.143 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 2021-03-09T08:47:26.143 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:47:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:47:26.144 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-03-09T08:47:26.144 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:47:26.144 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:47:26.144 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:47:26.510 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2021-03-09T08:47:26.511 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2021-03-09T08:47:26.512 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-03-09T08:47:26.620 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-03-09T08:47:26.649 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-03-09T08:47:26.714 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:26.714 INFO:tasks.ceph:role: osd.1 2021-03-09T08:47:26.714 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com 2021-03-09T08:47:26.715 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-09T08:47:26.785 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:47:26.785 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:47:26.786 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-03-09T08:47:26.786 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 2021-03-09T08:47:26.786 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:47:26.786 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:47:26.786 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-03-09T08:47:26.787 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:47:26.787 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:47:26.787 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:47:27.147 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2021-03-09T08:47:27.148 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2021-03-09T08:47:27.149 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-03-09T08:47:27.233 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-03-09T08:47:27.262 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-03-09T08:47:27.326 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:27.326 INFO:tasks.ceph:role: osd.2 2021-03-09T08:47:27.327 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com 2021-03-09T08:47:27.327 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-09T08:47:27.397 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:47:27.397 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:47:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-03-09T08:47:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 2021-03-09T08:47:27.398 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:47:27.398 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:47:27.398 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-03-09T08:47:27.399 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:47:27.399 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:47:27.399 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:47:27.761 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2021-03-09T08:47:27.763 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2021-03-09T08:47:27.764 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-03-09T08:47:27.854 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-03-09T08:47:27.882 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2021-03-09T08:47:27.946 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:27.947 INFO:tasks.ceph:role: osd.3 2021-03-09T08:47:27.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi099.front.sepia.ceph.com 2021-03-09T08:47:27.947 DEBUG:teuthology.orchestra.run.smithi099:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-03-09T08:47:28.018 INFO:teuthology.orchestra.run.smithi099.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:47:28.018 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:47:28.019 INFO:teuthology.orchestra.run.smithi099.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-03-09T08:47:28.019 INFO:teuthology.orchestra.run.smithi099.stdout: = reflink=1 2021-03-09T08:47:28.019 INFO:teuthology.orchestra.run.smithi099.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:47:28.019 INFO:teuthology.orchestra.run.smithi099.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:47:28.019 INFO:teuthology.orchestra.run.smithi099.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-03-09T08:47:28.020 INFO:teuthology.orchestra.run.smithi099.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:47:28.020 INFO:teuthology.orchestra.run.smithi099.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:47:28.020 INFO:teuthology.orchestra.run.smithi099.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:47:28.390 INFO:teuthology.orchestra.run.smithi099.stdout:Discarding blocks...Done. 2021-03-09T08:47:28.392 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi099.front.sepia.ceph.com -o noatime 2021-03-09T08:47:28.392 DEBUG:teuthology.orchestra.run.smithi099:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2021-03-09T08:47:28.509 DEBUG:teuthology.orchestra.run.smithi099:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2021-03-09T08:47:28.537 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:28.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:28.662+0000 7f2209bfb080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-03-09T08:47:28.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:28.663+0000 7f2209bfb080 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-03-09T08:47:28.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:28.663+0000 7f2209bfb080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T08:47:28.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:28.663+0000 7f2209bfb080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T08:47:28.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:28.663+0000 7f2209bfb080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-03-09T08:47:28.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:28.671+0000 7f2209bfb080 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-03-09T08:47:31.518 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-03-09T08:47:31.547 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:31.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:31.650+0000 7f58a04dd080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-03-09T08:47:31.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:31.651+0000 7f58a04dd080 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-03-09T08:47:31.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:31.651+0000 7f58a04dd080 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T08:47:31.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:31.651+0000 7f58a04dd080 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T08:47:31.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:31.651+0000 7f58a04dd080 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-03-09T08:47:31.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:31.651+0000 7f58a04dd080 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-03-09T08:47:34.496 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-03-09T08:47:34.532 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:34.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:34.643+0000 7fc553563080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-03-09T08:47:34.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:34.643+0000 7fc553563080 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-03-09T08:47:34.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:34.643+0000 7fc553563080 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T08:47:34.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:34.643+0000 7fc553563080 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T08:47:34.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:34.643+0000 7fc553563080 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-03-09T08:47:34.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:34.644+0000 7fc553563080 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-03-09T08:47:37.491 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-03-09T08:47:37.519 DEBUG:teuthology.orchestra.run.smithi099:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:37.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:37.625+0000 7f97c2039080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2021-03-09T08:47:37.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:37.626+0000 7f97c2039080 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2021-03-09T08:47:37.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:37.626+0000 7f97c2039080 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T08:47:37.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:37.626+0000 7f97c2039080 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T08:47:37.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:37.626+0000 7f97c2039080 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2021-03-09T08:47:37.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:37.626+0000 7f97c2039080 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2021-03-09T08:47:40.471 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2021-03-09T08:47:40.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2021-03-09T08:47:40.534 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:40.534 INFO:tasks.ceph:role: osd.4 2021-03-09T08:47:40.535 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com 2021-03-09T08:47:40.535 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-09T08:47:40.611 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:47:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:47:40.611 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-03-09T08:47:40.612 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 2021-03-09T08:47:40.612 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:47:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:47:40.613 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-03-09T08:47:40.613 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:47:40.613 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:47:40.614 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:47:40.634 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2021-03-09T08:47:40.636 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2021-03-09T08:47:40.636 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2021-03-09T08:47:40.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2021-03-09T08:47:40.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2021-03-09T08:47:40.863 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:40.864 INFO:tasks.ceph:role: osd.5 2021-03-09T08:47:40.864 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com 2021-03-09T08:47:40.865 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-09T08:47:40.929 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:47:40.929 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:47:40.929 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-03-09T08:47:40.930 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 2021-03-09T08:47:40.930 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:47:40.930 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:47:40.930 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-03-09T08:47:40.931 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:47:40.931 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:47:40.931 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:47:40.951 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2021-03-09T08:47:40.953 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2021-03-09T08:47:40.953 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2021-03-09T08:47:41.030 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2021-03-09T08:47:41.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2021-03-09T08:47:41.162 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:41.163 INFO:tasks.ceph:role: osd.6 2021-03-09T08:47:41.163 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com 2021-03-09T08:47:41.163 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-09T08:47:41.225 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:47:41.225 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:47:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-03-09T08:47:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 2021-03-09T08:47:41.226 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:47:41.226 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:47:41.227 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-03-09T08:47:41.227 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:47:41.227 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:47:41.227 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:47:41.249 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2021-03-09T08:47:41.250 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2021-03-09T08:47:41.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2021-03-09T08:47:41.327 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2021-03-09T08:47:41.395 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2021-03-09T08:47:41.458 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2021-03-09T08:47:41.459 INFO:tasks.ceph:role: osd.7 2021-03-09T08:47:41.459 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com 2021-03-09T08:47:41.460 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2021-03-09T08:47:41.522 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2021-03-09T08:47:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-09T08:47:41.522 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-03-09T08:47:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 2021-03-09T08:47:41.523 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-09T08:47:41.523 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2021-03-09T08:47:41.523 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-03-09T08:47:41.524 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-09T08:47:41.524 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-09T08:47:41.524 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-09T08:47:41.544 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2021-03-09T08:47:41.546 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2021-03-09T08:47:41.547 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2021-03-09T08:47:41.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2021-03-09T08:47:41.689 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:41.801 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:41.798+0000 7f8b8aacc080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2021-03-09T08:47:41.801 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:41.799+0000 7f8b8aacc080 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2021-03-09T08:47:41.802 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:41.799+0000 7f8b8aacc080 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-03-09T08:47:41.802 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:41.799+0000 7f8b8aacc080 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-03-09T08:47:41.802 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:41.799+0000 7f8b8aacc080 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2021-03-09T08:47:41.802 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:41.799+0000 7f8b8aacc080 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2021-03-09T08:47:44.643 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2021-03-09T08:47:44.673 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:44.785 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:44.783+0000 7fcd1d9a0080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2021-03-09T08:47:44.786 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:44.783+0000 7fcd1d9a0080 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2021-03-09T08:47:44.786 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:44.783+0000 7fcd1d9a0080 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-03-09T08:47:44.786 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:44.783+0000 7fcd1d9a0080 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-03-09T08:47:44.786 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:44.783+0000 7fcd1d9a0080 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2021-03-09T08:47:44.787 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:44.783+0000 7fcd1d9a0080 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2021-03-09T08:47:47.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-03-09T08:47:47.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:47.765 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:47.762+0000 7f52cb628080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2021-03-09T08:47:47.765 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:47.762+0000 7f52cb628080 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2021-03-09T08:47:47.766 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:47.762+0000 7f52cb628080 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-03-09T08:47:47.766 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:47.762+0000 7f52cb628080 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-03-09T08:47:47.766 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:47.762+0000 7f52cb628080 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2021-03-09T08:47:47.766 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:47.763+0000 7f52cb628080 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2021-03-09T08:47:50.607 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2021-03-09T08:47:50.636 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:50.744 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:50.742+0000 7f4dad057080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2021-03-09T08:47:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:50.742+0000 7f4dad057080 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2021-03-09T08:47:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:50.742+0000 7f4dad057080 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-03-09T08:47:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:50.742+0000 7f4dad057080 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-03-09T08:47:50.745 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:50.742+0000 7f4dad057080 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2021-03-09T08:47:50.746 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:47:50.742+0000 7f4dad057080 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2021-03-09T08:47:53.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2021-03-09T08:47:53.614 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-09T08:47:53.615 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:53.615 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2021-03-09T08:47:53.644 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:53.644 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2021-03-09T08:47:53.711 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:53.711 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2021-03-09T08:47:53.776 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:53.777 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2021-03-09T08:47:53.842 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:53.843 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2021-03-09T08:47:53.908 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:53.908 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2021-03-09T08:47:53.937 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:53.937 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2021-03-09T08:47:54.003 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:54.004 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2021-03-09T08:47:54.069 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:54.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2021-03-09T08:47:54.134 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:54.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2021-03-09T08:47:54.200 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:47:54.200 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2021-03-09T08:47:54.217 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:47:54.217 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2021-03-09T08:47:54.254 INFO:tasks.ceph:Adding keys to all mons... 2021-03-09T08:47:54.255 DEBUG:teuthology.orchestra.run.smithi099:> sudo tee -a /etc/ceph/ceph.keyring 2021-03-09T08:47:54.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo tee -a /etc/ceph/ceph.keyring 2021-03-09T08:47:54.298 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.y] 2021-03-09T08:47:54.298 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAdNkdgyf6KLhAAM0fThQoM33lUyMkiWebztQ== 2021-03-09T08:47:54.298 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.0] 2021-03-09T08:47:54.298 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAgNkdgbGCUJxAABUTpfhI2C0wZ4z4oQ65b9Q== 2021-03-09T08:47:54.299 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.1] 2021-03-09T08:47:54.299 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAjNkdgxUzeJhAAv1KJcbQO9jMobc72lOgkGA== 2021-03-09T08:47:54.299 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.2] 2021-03-09T08:47:54.299 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAmNkdgvg1tJhAAuJOqDmDSMnHgk9lu1I2H9g== 2021-03-09T08:47:54.300 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.3] 2021-03-09T08:47:54.300 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQApNkdgpVhgJRAALgi38c3ObsiRF0qU708lhg== 2021-03-09T08:47:54.300 INFO:teuthology.orchestra.run.smithi099.stdout:[mgr.x] 2021-03-09T08:47:54.300 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAdNkdgvW3fNRAALil8/wCqGFsr1Ldxx3vqOw== 2021-03-09T08:47:54.301 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.4] 2021-03-09T08:47:54.301 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAtNkdgXHKqLxAASEs2D41cuUE5SX+KzK43DA== 2021-03-09T08:47:54.302 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.5] 2021-03-09T08:47:54.302 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAwNkdgZtW4LhAAAmfP29feso6ZIReNx1SdYw== 2021-03-09T08:47:54.303 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.6] 2021-03-09T08:47:54.303 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAzNkdg3/d/LRAA4PiSBDxsy99Wgo2vM2IkAA== 2021-03-09T08:47:54.303 INFO:teuthology.orchestra.run.smithi099.stdout:[osd.7] 2021-03-09T08:47:54.303 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQA2NkdgGpBJLBAAkW6lPlNUTKBJrJNDtkkWpQ== 2021-03-09T08:47:54.304 INFO:teuthology.orchestra.run.smithi099.stdout:[client.0] 2021-03-09T08:47:54.304 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAdNkdgP59vORAARDIRNvZsoU7rGX3JKzcE8Q== 2021-03-09T08:47:54.304 INFO:teuthology.orchestra.run.smithi099.stdout:[client.1] 2021-03-09T08:47:54.304 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQAeNkdguS7xARAAXcigFw4d14mHz9m3tjdqEA== 2021-03-09T08:47:54.317 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.y] 2021-03-09T08:47:54.317 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAdNkdgyf6KLhAAM0fThQoM33lUyMkiWebztQ== 2021-03-09T08:47:54.318 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.0] 2021-03-09T08:47:54.318 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAgNkdgbGCUJxAABUTpfhI2C0wZ4z4oQ65b9Q== 2021-03-09T08:47:54.318 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.1] 2021-03-09T08:47:54.318 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAjNkdgxUzeJhAAv1KJcbQO9jMobc72lOgkGA== 2021-03-09T08:47:54.319 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.2] 2021-03-09T08:47:54.319 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAmNkdgvg1tJhAAuJOqDmDSMnHgk9lu1I2H9g== 2021-03-09T08:47:54.319 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.3] 2021-03-09T08:47:54.319 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQApNkdgpVhgJRAALgi38c3ObsiRF0qU708lhg== 2021-03-09T08:47:54.320 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr.x] 2021-03-09T08:47:54.320 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAdNkdgvW3fNRAALil8/wCqGFsr1Ldxx3vqOw== 2021-03-09T08:47:54.320 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.4] 2021-03-09T08:47:54.320 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAtNkdgXHKqLxAASEs2D41cuUE5SX+KzK43DA== 2021-03-09T08:47:54.321 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.5] 2021-03-09T08:47:54.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAwNkdgZtW4LhAAAmfP29feso6ZIReNx1SdYw== 2021-03-09T08:47:54.321 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.6] 2021-03-09T08:47:54.321 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAzNkdg3/d/LRAA4PiSBDxsy99Wgo2vM2IkAA== 2021-03-09T08:47:54.321 INFO:teuthology.orchestra.run.smithi195.stdout:[osd.7] 2021-03-09T08:47:54.322 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQA2NkdgGpBJLBAAkW6lPlNUTKBJrJNDtkkWpQ== 2021-03-09T08:47:54.322 INFO:teuthology.orchestra.run.smithi195.stdout:[client.0] 2021-03-09T08:47:54.322 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAdNkdgP59vORAARDIRNvZsoU7rGX3JKzcE8Q== 2021-03-09T08:47:54.322 INFO:teuthology.orchestra.run.smithi195.stdout:[client.1] 2021-03-09T08:47:54.323 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQAeNkdguS7xARAAXcigFw4d14mHz9m3tjdqEA== 2021-03-09T08:47:54.324 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:47:54.349 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:47:54.428 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.430 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.495 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.497 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.560 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.563 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.627 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.630 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.695 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:47:54.697 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-09T08:47:54.761 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.763 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.827 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.829 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.959 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:54.962 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-09T08:47:55.026 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T08:47:55.029 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T08:47:55.095 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T08:47:55.098 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-09T08:47:55.164 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-09T08:47:55.165 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-03-09T08:47:55.188 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T08:47:55.297 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-03-09T08:47:55.321 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-03-09T08:47:55.384 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T08:47:55.492 DEBUG:teuthology.orchestra.run.smithi099:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-03-09T08:47:55.516 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-03-09T08:47:55.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-03-09T08:47:55.654 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-03-09T08:47:55.678 DEBUG:teuthology.orchestra.run.smithi099:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:55.680 DEBUG:teuthology.orchestra.run.smithi195:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-03-09T08:47:55.732 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-03-09T08:47:55.732 INFO:tasks.ceph.mon.a:Restarting daemon 2021-03-09T08:47:55.733 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-03-09T08:47:55.735 INFO:tasks.ceph.mon.a:Started 2021-03-09T08:47:55.736 INFO:tasks.ceph.mon.c:Restarting daemon 2021-03-09T08:47:55.736 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-03-09T08:47:55.738 INFO:tasks.ceph.mon.c:Started 2021-03-09T08:47:55.739 INFO:tasks.ceph.mon.b:Restarting daemon 2021-03-09T08:47:55.739 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-03-09T08:47:55.773 INFO:tasks.ceph.mon.b:Started 2021-03-09T08:47:55.774 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-03-09T08:47:55.774 INFO:tasks.ceph.mgr.y:Restarting daemon 2021-03-09T08:47:55.775 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2021-03-09T08:47:55.777 INFO:tasks.ceph.mgr.y:Started 2021-03-09T08:47:55.777 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-03-09T08:47:55.778 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-03-09T08:47:55.780 INFO:tasks.ceph.mgr.x:Started 2021-03-09T08:47:55.780 INFO:tasks.ceph:Setting crush tunables to default 2021-03-09T08:47:55.780 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster ceph osd crush tunables default 2021-03-09T08:47:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.917+0000 7f91a81ad700 10 monclient: get_monmap_and_config 2021-03-09T08:47:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.917+0000 7f91a81ad700 10 monclient: build_initial_monmap 2021-03-09T08:47:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.917+0000 7f91a81ad700 10 monclient: monmap: 2021-03-09T08:47:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 0 2021-03-09T08:47:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:47:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:55.919069+0000 2021-03-09T08:47:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:55.919069+0000 2021-03-09T08:47:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 0 (unknown) 2021-03-09T08:47:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:47:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:47:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:47:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:47:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:47:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:47:55.923 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:47:55.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.918+0000 7f91a81ad700 1 Processor -- start 2021-03-09T08:47:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.918+0000 7f91a81ad700 1 -- start start 2021-03-09T08:47:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.918+0000 7f91a81ad700 10 monclient: init 2021-03-09T08:47:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 10 monclient: _reopen_session rank -1 2021-03-09T08:47:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2021-03-09T08:47:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:47:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 10 monclient(hunting): picked mon.noname-b con 0x7f91a00ada30 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:47:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91a00a5930 addr v1:172.21.15.195:6789/0 2021-03-09T08:47:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91a0138f80 addr v1:172.21.15.99:6789/0 2021-03-09T08:47:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a00ade10 con 0x7f91a0138f80 2021-03-09T08:47:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a00a8dd0 con 0x7f91a00a5930 2021-03-09T08:47:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 10 monclient(hunting): start opening mon connection 2021-03-09T08:47:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00a8ac0 con 0x7f91a00ada30 2021-03-09T08:47:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 10 monclient(hunting): _renew_subs 2021-03-09T08:47:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a81ad700 10 monclient(hunting): authenticate will time out at 842.394268s 2021-03-09T08:47:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a674a700 1 --1- >> v1:172.21.15.99:6789/0 conn(0x7f91a0138f80 0x7f91a0139410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.99:6789/0 says I am v1:172.21.15.99:32854/0 (socket says 172.21.15.99:32854) 2021-03-09T08:47:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a674a700 1 -- 172.21.15.99:0/1269937289 learned_addr learned my addr 172.21.15.99:0/1269937289 (peer_addr_for_me v1:172.21.15.99:0/0) 2021-03-09T08:47:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f91a00ada30 auth_method 0 2021-03-09T08:47:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 msgr2=0x7f91a00a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:47:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 msgr2=0x7f91a00a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:55.919+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:47:56.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.120+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:56.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.120+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f91a00ada30 auth_method 0 2021-03-09T08:47:56.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.120+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:56.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.120+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:56.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.120+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 msgr2=0x7f91a00a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:47:56.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.120+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 msgr2=0x7f91a00a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:56.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.120+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:56.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.120+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:47:56.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.521+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:56.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.521+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f91a00ada30 auth_method 0 2021-03-09T08:47:56.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.521+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:56.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.521+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:56.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.521+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 msgr2=0x7f91a00a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:47:56.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.521+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 msgr2=0x7f91a00a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:56.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.521+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:56.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:56.521+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:47:57.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:57.322+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:57.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:57.322+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f91a00ada30 auth_method 0 2021-03-09T08:47:57.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:57.322+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:57.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:57.322+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:57.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:57.322+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 msgr2=0x7f91a00a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:47:57.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:57.322+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 msgr2=0x7f91a00a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:57.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:57.322+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:57.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:57.322+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:47:58.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 10 monclient(hunting): tick 2021-03-09T08:47:58.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 1 monclient(hunting): continuing hunt 2021-03-09T08:47:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T08:47:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 msgr2=0x7f91a00a6be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:47:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f91a00ada30 0x7f91a00a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:47:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 >> v1:172.21.15.195:6789/0 conn(0x7f91a00a5930 legacy=0x7f91a00a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:47:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 >> v1:172.21.15.99:6789/0 conn(0x7f91a0138f80 legacy=0x7f91a0139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:47:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 10 monclient: _add_conns ranks=[3,4] 2021-03-09T08:47:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:47:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 10 monclient(hunting): picked mon.noname-a con 0x7f9180003430 addr v2:172.21.15.99:3300/0 2021-03-09T08:47:59.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:47:59.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 10 monclient(hunting): picked mon.noname-c con 0x7f9180005f30 addr v2:172.21.15.195:3300/0 2021-03-09T08:47:59.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 10 monclient(hunting): start opening mon connection 2021-03-09T08:47:59.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a00a8ac0 con 0x7f9180003430 2021-03-09T08:47:59.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 10 monclient(hunting): start opening mon connection 2021-03-09T08:47:59.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9180008a60 con 0x7f9180005f30 2021-03-09T08:47:59.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91977fe700 10 monclient(hunting): _renew_subs 2021-03-09T08:47:59.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:59.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f9180003430 auth_method 0 2021-03-09T08:47:59.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 msgr2=0x7f9180005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:47:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 msgr2=0x7f9180005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:47:59.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:59.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5748700 10 monclient(hunting): get_auth_request con 0x7f9180005f30 auth_method 0 2021-03-09T08:47:59.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5748700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5748700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 msgr2=0x7f91800083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:47:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 msgr2=0x7f91800083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:59.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:58.919+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:47:59.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:59.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f9180003430 auth_method 0 2021-03-09T08:47:59.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:59.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:59.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 msgr2=0x7f9180005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:47:59.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 msgr2=0x7f9180005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:59.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:59.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:47:59.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:59.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5748700 10 monclient(hunting): get_auth_request con 0x7f9180005f30 auth_method 0 2021-03-09T08:47:59.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5748700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:59.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5748700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:59.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 msgr2=0x7f91800083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:47:59.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 msgr2=0x7f91800083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:59.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:59.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.120+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:47:59.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:59.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f9180003430 auth_method 0 2021-03-09T08:47:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 msgr2=0x7f9180005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:47:59.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 msgr2=0x7f9180005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:47:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:47:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5748700 10 monclient(hunting): get_auth_request con 0x7f9180005f30 auth_method 0 2021-03-09T08:47:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5748700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:47:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5748700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:47:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 msgr2=0x7f91800083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:47:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 msgr2=0x7f91800083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:47:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:47:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:47:59.521+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:48:00.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:00.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f9180003430 auth_method 0 2021-03-09T08:48:00.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:00.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:00.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 msgr2=0x7f9180005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:48:00.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 msgr2=0x7f9180005890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:48:00.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:48:00.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:48:00.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:00.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5748700 10 monclient(hunting): get_auth_request con 0x7f9180005f30 auth_method 0 2021-03-09T08:48:00.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5748700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:00.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5748700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:00.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 msgr2=0x7f91800083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2021-03-09T08:48:00.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 msgr2=0x7f91800083f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:48:00.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:48:00.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:00.322+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:48:01.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 10 monclient(hunting): tick 2021-03-09T08:48:01.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 monclient(hunting): continuing hunt 2021-03-09T08:48:01.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T08:48:01.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 msgr2=0x7f91800083f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:01.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180005f30 0x7f91800083f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:01.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 msgr2=0x7f9180005890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:01.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180003430 0x7f9180005890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:01.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2021-03-09T08:48:01.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91a674a700 1 -- 172.21.15.99:0/1269937289 reap_dead start 2021-03-09T08:48:01.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180008e10 0x7f918000b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:01.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 10 monclient(hunting): picked mon.noname-c con 0x7f9180008e10 addr v2:172.21.15.195:3300/0 2021-03-09T08:48:01.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9180003880 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:01.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180007b30 0x7f9180007fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:01.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 10 monclient(hunting): picked mon.noname-a con 0x7f9180007b30 addr v2:172.21.15.99:3300/0 2021-03-09T08:48:01.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a00a4480 con 0x7f9180003880 2021-03-09T08:48:01.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:01.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a00a8ac0 con 0x7f9180007b30 2021-03-09T08:48:01.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:01.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 1 -- 172.21.15.99:0/1269937289 --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9180008a60 con 0x7f9180008e10 2021-03-09T08:48:01.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.919+0000 7f91977fe700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:01.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a674a700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180007b30 0x7f9180007fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:01.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a674a700 10 monclient(hunting): get_auth_request con 0x7f9180007b30 auth_method 0 2021-03-09T08:48:01.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a674a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:01.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a674a700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:01.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a674a700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180007b30 msgr2=0x7f9180007fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T08:48:01.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a674a700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180007b30 msgr2=0x7f9180007fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:48:01.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180008e10 0x7f918000b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:01.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a674a700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180007b30 0x7f9180007fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:48:01.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f9180008e10 auth_method 0 2021-03-09T08:48:01.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:01.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:01.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a674a700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180007b30 0x7f9180007fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:01.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180008e10 msgr2=0x7f918000b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2021-03-09T08:48:01.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a5f49700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180008e10 msgr2=0x7f918000b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:48:01.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180008e10 0x7f918000b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:48:01.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:01.920+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180008e10 0x7f918000b200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:02.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.085+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076403019 0 0) 0x7f91a00a4480 con 0x7f9180003880 2021-03-09T08:48:02.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.085+0000 7f91a4f47700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.085+0000 7f91a4f47700 10 monclient(hunting): my global_id is 4102 2021-03-09T08:48:02.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.085+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f918c003270 con 0x7f9180003880 2021-03-09T08:48:02.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 710549246 0 0) 0x7f918c003270 con 0x7f9180003880 2021-03-09T08:48:02.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180008e10 msgr2=0x7f918000b200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:02.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.195:3300/0 conn(0x7f9180008e10 0x7f918000b200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180007b30 msgr2=0x7f9180007fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:02.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 --2- 172.21.15.99:0/1269937289 >> v2:172.21.15.99:3300/0 conn(0x7f9180007b30 0x7f9180007fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient: _finish_hunting 0 2021-03-09T08:48:02.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 monclient: found mon.noname-c-legacy 2021-03-09T08:48:02.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:02.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.195:6789/0 2021-03-09T08:48:02.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91800086c0 con 0x7f9180003880 2021-03-09T08:48:02.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient: _finish_auth 0 2021-03-09T08:48:02.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:02.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1912569104 0 0) 0x7f9190007b80 con 0x7f9180003880 2021-03-09T08:48:02.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:02.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2021-03-09T08:48:02.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient: dump: 2021-03-09T08:48:02.097 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:02.097 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.097 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.097 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.097 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:02.098 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:02.098 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:02.098 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:02.098 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:02.099 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:02.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] but i'm connected to v1:172.21.15.195:6789/0, reconnecting 2021-03-09T08:48:02.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:02.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 >> v1:172.21.15.195:6789/0 conn(0x7f9180003880 legacy=0x7f91800063e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:48:02.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f918c003760 0x7f918c003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient(hunting): picked mon.a con 0x7f918c003760 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:02.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918c0040e0 0x7f918c0045b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient(hunting): picked mon.b con 0x7f918c0040e0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:02.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c004c50 0x7f918c00b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient(hunting): picked mon.c con 0x7f918c004c50 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:02.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00a8ac0 con 0x7f918c003760 2021-03-09T08:48:02.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9180008a60 con 0x7f918c0040e0 2021-03-09T08:48:02.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918c005370 con 0x7f918c004c50 2021-03-09T08:48:02.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a4f47700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:02.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a674a700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c004c50 0x7f918c00b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a674a700 10 monclient(hunting): get_auth_request con 0x7f918c004c50 auth_method 0 2021-03-09T08:48:02.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a674a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a674a700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f918c003760 0x7f918c003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a674a700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a674a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918c0040e0 0x7f918c0045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a674a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5748700 10 monclient(hunting): get_auth_request con 0x7f918c0040e0 auth_method 0 2021-03-09T08:48:02.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5748700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5748700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f918c003760 auth_method 0 2021-03-09T08:48:02.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5748700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5748700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.086+0000 7f91a5748700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 10 monclient(hunting): handle_auth_done global_id 4102 payload 450 2021-03-09T08:48:02.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c004c50 msgr2=0x7f918c00b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:48:02.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c004c50 0x7f918c00b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f918c003760 msgr2=0x7f918c003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:48:02.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f918c003760 0x7f918c003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 10 monclient: _finish_hunting 0 2021-03-09T08:48:02.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 1 monclient: found mon.b 2021-03-09T08:48:02.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:02.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:02.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 1 -- 172.21.15.99:0/1269937289 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f918c003270 con 0x7f918c0040e0 2021-03-09T08:48:02.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 10 monclient: _finish_auth 0 2021-03-09T08:48:02.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:02.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a674a700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c004c50 0x7f918c00b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T08:48:02.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a674a700 1 -- 172.21.15.99:0/1269937289 reap_dead start 2021-03-09T08:48:02.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a81ad700 5 monclient: authenticate success, global_id 4102 2021-03-09T08:48:02.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a81ad700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:02.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a5748700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918c0040e0 0x7f918c0045b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f91800089d0 tx=0x7f9190009040).ready entity=mon.1 client_cookie=b327e426334a29f1 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:02.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91900034d0 con 0x7f918c0040e0 2021-03-09T08:48:02.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:02.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:02.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 10 monclient: dump: 2021-03-09T08:48:02.111 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:02.111 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:02.112 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:02.113 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:02.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a81ad700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190014d60 con 0x7f918c0040e0 2021-03-09T08:48:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a81ad700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 1 -- 172.21.15.99:0/1269937289 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190017690 con 0x7f918c0040e0 2021-03-09T08:48:02.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a81ad700 4 set_mon_vals no callback set 2021-03-09T08:48:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.087+0000 7f91a4f47700 10 monclient: dump: 2021-03-09T08:48:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:02.115 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.116 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.116 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.116 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:02.116 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:02.116 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:02.117 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:02.117 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:02.117 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:02.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.088+0000 7f91a81ad700 10 monclient: shutdown 2021-03-09T08:48:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.088+0000 7f91a81ad700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.088+0000 7f91a81ad700 1 -- 172.21.15.99:0/1269937289 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918c0040e0 msgr2=0x7f918c0045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.088+0000 7f91a81ad700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918c0040e0 0x7f918c0045b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f91800089d0 tx=0x7f9190009040).stop 2021-03-09T08:48:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.088+0000 7f91a81ad700 1 -- 172.21.15.99:0/1269937289 shutdown_connections 2021-03-09T08:48:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.088+0000 7f91a81ad700 1 --2- 172.21.15.99:0/1269937289 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f918c0040e0 0x7f918c0045b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.088+0000 7f91a81ad700 1 -- 172.21.15.99:0/1269937289 >> 172.21.15.99:0/1269937289 conn(0x7f91a00631d0 msgr2=0x7f91a00635b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.088+0000 7f91a81ad700 1 -- 172.21.15.99:0/1269937289 shutdown_connections 2021-03-09T08:48:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.088+0000 7f91a81ad700 1 -- 172.21.15.99:0/1269937289 wait complete. 2021-03-09T08:48:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.089+0000 7f91a81ad700 10 monclient: build_initial_monmap 2021-03-09T08:48:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.089+0000 7f91a81ad700 10 monclient: monmap: 2021-03-09T08:48:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 0 2021-03-09T08:48:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 0.000000 2021-03-09T08:48:02.121 INFO:teuthology.orchestra.run.smithi099.stderr:created 0.000000 2021-03-09T08:48:02.121 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:02.121 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 1 2021-03-09T08:48:02.122 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:02.122 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:02.122 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:02.122 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.089+0000 7f91a81ad700 1 librados: starting msgr at 2021-03-09T08:48:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.089+0000 7f91a81ad700 1 librados: starting objecter 2021-03-09T08:48:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.089+0000 7f91a81ad700 1 Processor -- start 2021-03-09T08:48:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.089+0000 7f91a81ad700 1 -- start start 2021-03-09T08:48:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.089+0000 7f91a81ad700 1 librados: setting wanted keys 2021-03-09T08:48:02.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.089+0000 7f91a81ad700 1 librados: calling monclient init 2021-03-09T08:48:02.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.089+0000 7f91a81ad700 10 monclient: init 2021-03-09T08:48:02.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:02.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:48:02.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c0040e0 0x7f91a011b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient(hunting): picked mon.noname-c con 0x7f918c0040e0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:02.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f91a01370f0 0x7f91a0145c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient(hunting): picked mon.noname-a con 0x7f91a01370f0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:02.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91a0137570 0x7f91a0118940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient(hunting): picked mon.noname-b con 0x7f91a0137570 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:02.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a01466c0 con 0x7f91a01370f0 2021-03-09T08:48:02.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a0146b60 con 0x7f91a0137570 2021-03-09T08:48:02.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a01175b0 con 0x7f918c0040e0 2021-03-09T08:48:02.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:02.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a81ad700 10 monclient(hunting): authenticate will time out at 848.565923s 2021-03-09T08:48:02.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.090+0000 7f91a5748700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f91a01370f0 0x7f91a0145c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f91a01370f0 0x7f91a0145c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33790/0 (socket says 172.21.15.99:33790) 2021-03-09T08:48:02.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 1 -- 172.21.15.99:0/1383635352 learned_addr learned my addr 172.21.15.99:0/1383635352 (peer_addr_for_me v2:172.21.15.99:0/0) 2021-03-09T08:48:02.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 10 monclient(hunting): get_auth_request con 0x7f91a01370f0 auth_method 0 2021-03-09T08:48:02.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5f49700 1 --2- 172.21.15.99:0/1383635352 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c0040e0 0x7f91a011b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5f49700 10 monclient(hunting): get_auth_request con 0x7f918c0040e0 auth_method 0 2021-03-09T08:48:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5f49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5f49700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a674a700 1 --2- 172.21.15.99:0/1383635352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91a0137570 0x7f91a0118940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5f49700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5f49700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5f49700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a674a700 10 monclient(hunting): get_auth_request con 0x7f91a0137570 auth_method 0 2021-03-09T08:48:02.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a674a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a674a700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a674a700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a674a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a674a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 10 monclient(hunting): handle_auth_done global_id 4110 payload 995 2021-03-09T08:48:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 1 -- 172.21.15.99:0/1383635352 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c0040e0 msgr2=0x7f91a011b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 1 --2- 172.21.15.99:0/1383635352 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c0040e0 0x7f91a011b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 1 -- 172.21.15.99:0/1383635352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91a0137570 msgr2=0x7f91a0118940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 1 --2- 172.21.15.99:0/1383635352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91a0137570 0x7f91a0118940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 10 monclient: _finish_hunting 0 2021-03-09T08:48:02.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 1 monclient: found mon.noname-a 2021-03-09T08:48:02.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:02.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.99:3300/0 2021-03-09T08:48:02.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 1 -- 172.21.15.99:0/1383635352 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a011b650 con 0x7f91a01370f0 2021-03-09T08:48:02.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f91a5748700 1 --2- 172.21.15.99:0/1383635352 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f91a01370f0 0x7f91a0145c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f918c005340 tx=0x7f9190009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:02.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f9195ffb700 10 client.?.objecter ms_handle_connect 0x7f91a01370f0 2021-03-09T08:48:02.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.091+0000 7f9195ffb700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:02.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 1 -- 172.21.15.99:0/1383635352 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190007b80 con 0x7f91a01370f0 2021-03-09T08:48:02.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-03-09T08:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 10 monclient: dump: 2021-03-09T08:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.143 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.143 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:02.143 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:02.143 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:02.143 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:02.144 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:02.144 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:02.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 10 monclient: _finish_auth 0 2021-03-09T08:48:02.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:02.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 1 -- 172.21.15.99:0/1383635352 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190014ed0 con 0x7f91a01370f0 2021-03-09T08:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f91a81ad700 5 monclient: authenticate success, global_id 4110 2021-03-09T08:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 1 -- 172.21.15.99:0/1383635352 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91900075b0 con 0x7f91a01370f0 2021-03-09T08:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f91a81ad700 10 monclient: _renew_subs 2021-03-09T08:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f91a81ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.99:3300/0 2021-03-09T08:48:02.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f91a81ad700 1 -- 172.21.15.99:0/1383635352 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a0146180 con 0x7f91a01370f0 2021-03-09T08:48:02.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9197fff700 4 set_mon_vals no callback set 2021-03-09T08:48:02.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:02.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-03-09T08:48:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f9195ffb700 10 monclient: dump: 2021-03-09T08:48:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:02.149 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:02.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f91a81ad700 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:02.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f91a81ad700 10 monclient: _renew_subs 2021-03-09T08:48:02.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f91a81ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.99:3300/0 2021-03-09T08:48:02.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f91a81ad700 1 -- 172.21.15.99:0/1383635352 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a0119070 con 0x7f91a01370f0 2021-03-09T08:48:02.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.092+0000 7f91a81ad700 1 librados: init done 2021-03-09T08:48:02.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f9195ffb700 1 -- 172.21.15.99:0/1383635352 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f9190032070 con 0x7f91a01370f0 2021-03-09T08:48:02.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f9195ffb700 1 -- 172.21.15.99:0/1383635352 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9190022780 con 0x7f91a01370f0 2021-03-09T08:48:02.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f9195ffb700 10 client.4110.objecter ms_dispatch 0x7f91a0001550 osd_map(1..1 src has 1..1) v4 2021-03-09T08:48:02.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f9195ffb700 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2021-03-09T08:48:02.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f9195ffb700 3 client.4110.objecter handle_osd_map decoding full epoch 1 2021-03-09T08:48:02.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f9195ffb700 20 client.4110.objecter dump_active .. 0 homeless 2021-03-09T08:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f91a81ad700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f91a81ad700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f91a81ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.99:3300/0 2021-03-09T08:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.093+0000 7f91a81ad700 1 -- 172.21.15.99:0/1383635352 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a011b7e0 con 0x7f91a01370f0 2021-03-09T08:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.096+0000 7f9195ffb700 1 -- 172.21.15.99:0/1383635352 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f919004a980 con 0x7f91a01370f0 2021-03-09T08:48:02.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.096+0000 7f9195ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:02.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.096+0000 7f9195ffb700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:02.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.170+0000 7f91a81ad700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2021-03-09T08:48:02.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.170+0000 7f91a81ad700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2021-03-09T08:48:02.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.170+0000 7f91a81ad700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.99:3300/0 2021-03-09T08:48:02.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.170+0000 7f91a81ad700 1 -- 172.21.15.99:0/1383635352 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f91a00a68a0 con 0x7f91a01370f0 2021-03-09T08:48:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.173+0000 7f9195ffb700 1 -- 172.21.15.99:0/1383635352 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9190039070 con 0x7f91a01370f0 2021-03-09T08:48:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.173+0000 7f9195ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2021-03-09T08:48:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.173+0000 7f9195ffb700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2021-03-09T08:48:02.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 10 librados: watch_flush enter 2021-03-09T08:48:02.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 10 librados: watch_flush exit 2021-03-09T08:48:02.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 20 client.4110.objecter shutdown clearing up homeless session... 2021-03-09T08:48:02.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 10 client.4110.objecter successfully canceled tick 2021-03-09T08:48:02.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 10 monclient: shutdown 2021-03-09T08:48:02.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 -- 172.21.15.99:0/1383635352 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f91a01370f0 msgr2=0x7f91a0145c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 --2- 172.21.15.99:0/1383635352 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f91a01370f0 0x7f91a0145c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f918c005340 tx=0x7f9190009040).stop 2021-03-09T08:48:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 -- 172.21.15.99:0/1383635352 shutdown_connections 2021-03-09T08:48:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 --2- 172.21.15.99:0/1383635352 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f91a0137570 0x7f91a0118940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 --2- 172.21.15.99:0/1383635352 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f918c0040e0 0x7f91a011b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 --2- 172.21.15.99:0/1383635352 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f91a01370f0 0x7f91a0145c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 -- 172.21.15.99:0/1383635352 >> 172.21.15.99:0/1383635352 conn(0x7f91a00631d0 msgr2=0x7f91a00635b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 -- 172.21.15.99:0/1383635352 shutdown_connections 2021-03-09T08:48:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 -- 172.21.15.99:0/1383635352 wait complete. 2021-03-09T08:48:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:02.174+0000 7f91a81ad700 1 librados: shutdown 2021-03-09T08:48:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:adjusted tunables profile to default 2021-03-09T08:48:02.189 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-03-09T08:48:02.189 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:48:02.189 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2021-03-09T08:48:02.214 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:48:02.214 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2021-03-09T08:48:02.278 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:48:02.278 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2021-03-09T08:48:02.342 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2021-03-09T08:48:02.343 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2021-03-09T08:48:02.406 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:48:02.407 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2021-03-09T08:48:02.430 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:48:02.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2021-03-09T08:48:02.494 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:48:02.495 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2021-03-09T08:48:02.558 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2021-03-09T08:48:02.559 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2021-03-09T08:48:02.623 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new bfcce70f-6a3c-4d76-9a45-145c7d5e468d 0 2021-03-09T08:48:02.791 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.789+0000 7f50ee48f700 10 monclient: get_monmap_and_config 2021-03-09T08:48:02.791 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.789+0000 7f50ee48f700 10 monclient: build_initial_monmap 2021-03-09T08:48:02.791 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.789+0000 7f50ee48f700 10 monclient: monmap: 2021-03-09T08:48:02.792 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:02.792 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.792 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:48:02.790167+0000 2021-03-09T08:48:02.792 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:48:02.790167+0000 2021-03-09T08:48:02.793 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:02.793 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:02.793 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:02.793 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:02.793 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:02.794 INFO:teuthology.orchestra.run.smithi195.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:02.794 INFO:teuthology.orchestra.run.smithi195.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:02.794 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 1 Processor -- start 2021-03-09T08:48:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 1 -- start start 2021-03-09T08:48:02.805 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient: init 2021-03-09T08:48:02.806 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:02.806 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2021-03-09T08:48:02.806 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f50e80a5fd0 0x7f50e80a4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.806 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient(hunting): picked mon.noname-a con 0x7f50e80a5fd0 addr v2:172.21.15.99:3300/0 2021-03-09T08:48:02.807 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50e80a3950 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:02.807 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e81304f0 0x7f50e8133090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.807 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient(hunting): picked mon.noname-b con 0x7f50e81304f0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:02.808 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e809e450 con 0x7f50e80a3950 2021-03-09T08:48:02.808 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.808 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e80a4990 con 0x7f50e80a5fd0 2021-03-09T08:48:02.808 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.809 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e80a63b0 con 0x7f50e81304f0 2021-03-09T08:48:02.809 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:02.809 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.790+0000 7f50ee48f700 10 monclient(hunting): authenticate will time out at 848.280855s 2021-03-09T08:48:02.809 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ecc8c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f50e80a3950 0x7f50e812fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47974/0 (socket says 172.21.15.195:47974) 2021-03-09T08:48:02.810 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ecc8c700 1 -- 172.21.15.195:0/1012722836 learned_addr learned my addr 172.21.15.195:0/1012722836 (peer_addr_for_me v1:172.21.15.195:0/0) 2021-03-09T08:48:02.810 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 1 --2- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e81304f0 0x7f50e8133090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.810 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 1 --2- 172.21.15.195:0/1012722836 >> v2:172.21.15.99:3300/0 conn(0x7f50e80a5fd0 0x7f50e80a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.810 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 10 monclient(hunting): get_auth_request con 0x7f50e81304f0 auth_method 0 2021-03-09T08:48:02.810 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.811 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.811 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 10 monclient(hunting): get_auth_request con 0x7f50e80a5fd0 auth_method 0 2021-03-09T08:48:02.811 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.812 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 1 -- 172.21.15.195:0/1012722836 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984978516 0 0) 0x7f50e809e450 con 0x7f50e80a3950 2021-03-09T08:48:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient(hunting): my global_id is 4114 2021-03-09T08:48:02.814 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 1 -- 172.21.15.195:0/1012722836 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50d0003270 con 0x7f50e80a3950 2021-03-09T08:48:02.814 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 1 -- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e81304f0 msgr2=0x7f50e8133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2021-03-09T08:48:02.814 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 1 -- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e81304f0 msgr2=0x7f50e8133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2021-03-09T08:48:02.814 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 1 --2- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e81304f0 0x7f50e8133090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:48:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50edc8e700 1 --2- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e81304f0 0x7f50e8133090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 10 monclient(hunting): handle_auth_done global_id 4113 payload 450 2021-03-09T08:48:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 1 -- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e81304f0 msgr2=0x7f50e8133090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 1 --2- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e81304f0 0x7f50e8133090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.815 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 1 -- 172.21.15.195:0/1012722836 >> v1:172.21.15.195:6789/0 conn(0x7f50e80a3950 legacy=0x7f50e812fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.816 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 10 monclient: _finish_hunting 0 2021-03-09T08:48:02.816 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 1 monclient: found mon.noname-a 2021-03-09T08:48:02.816 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:02.816 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.99:3300/0 2021-03-09T08:48:02.817 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 1 -- 172.21.15.195:0/1012722836 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e80a4730 con 0x7f50e80a5fd0 2021-03-09T08:48:02.817 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ed48d700 1 --2- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50e80a5fd0 0x7f50e80a4240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f50dc00b9c0 tx=0x7f50dc006e20).ready entity=mon.0 client_cookie=1ab57ddc97ded5ef server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:02.817 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 1 -- 172.21.15.195:0/1012722836 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc009600 con 0x7f50e80a5fd0 2021-03-09T08:48:02.817 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:02.817 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-03-09T08:48:02.818 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient: dump: 2021-03-09T08:48:02.818 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:02.818 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.818 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.818 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.819 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:02.819 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:02.819 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:02.819 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:02.819 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:02.820 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:02.820 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient: _finish_auth 0 2021-03-09T08:48:02.820 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:02.820 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ee48f700 5 monclient: authenticate success, global_id 4113 2021-03-09T08:48:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 1 -- 172.21.15.195:0/1012722836 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50dc009760 con 0x7f50e80a5fd0 2021-03-09T08:48:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ee48f700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ee48f700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:02.821 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50ee48f700 4 set_mon_vals no callback set 2021-03-09T08:48:02.822 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 1 -- 172.21.15.195:0/1012722836 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50dc009ae0 con 0x7f50e80a5fd0 2021-03-09T08:48:02.822 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:02.822 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-03-09T08:48:02.822 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.791+0000 7f50d7fff700 10 monclient: dump: 2021-03-09T08:48:02.822 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:02.823 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:02.824 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:02.824 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:02.824 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:02.824 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:02.824 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 10 monclient: shutdown 2021-03-09T08:48:02.825 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:02.825 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 1 -- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50e80a5fd0 msgr2=0x7f50e80a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.825 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 1 --2- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50e80a5fd0 0x7f50e80a4240 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f50dc00b9c0 tx=0x7f50dc006e20).stop 2021-03-09T08:48:02.825 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 1 -- 172.21.15.195:0/1012722836 shutdown_connections 2021-03-09T08:48:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 1 --2- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e81304f0 0x7f50e8133090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 1 --2- 172.21.15.195:0/1012722836 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50e80a5fd0 0x7f50e80a4240 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 1 -- 172.21.15.195:0/1012722836 >> 172.21.15.195:0/1012722836 conn(0x7f50e80642c0 msgr2=0x7f50e8069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 1 -- 172.21.15.195:0/1012722836 shutdown_connections 2021-03-09T08:48:02.826 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 1 -- 172.21.15.195:0/1012722836 wait complete. 2021-03-09T08:48:02.827 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 10 monclient: build_initial_monmap 2021-03-09T08:48:02.827 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.792+0000 7f50ee48f700 10 monclient: monmap: 2021-03-09T08:48:02.827 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:02.827 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:02.827 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2021-03-09T08:48:02.828 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2021-03-09T08:48:02.828 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:02.828 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2021-03-09T08:48:02.828 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:02.828 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:02.829 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:02.829 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:02.829 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 librados: starting msgr at 2021-03-09T08:48:02.829 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 librados: starting objecter 2021-03-09T08:48:02.830 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 Processor -- start 2021-03-09T08:48:02.830 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 -- start start 2021-03-09T08:48:02.830 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 librados: setting wanted keys 2021-03-09T08:48:02.830 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 librados: calling monclient init 2021-03-09T08:48:02.830 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient: init 2021-03-09T08:48:02.831 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:02.831 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:48:02.831 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e80a3950 0x7f50e80681d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.831 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient(hunting): picked mon.noname-c con 0x7f50e80a3950 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:02.831 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50e80a5fd0 0x7f50e8141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.832 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient(hunting): picked mon.noname-b con 0x7f50e80a5fd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:02.832 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50e81304f0 0x7f50e8142110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:02.832 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient(hunting): picked mon.noname-a con 0x7f50e81304f0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:02.832 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.833 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8136430 con 0x7f50e81304f0 2021-03-09T08:48:02.833 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.833 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8067d30 con 0x7f50e80a5fd0 2021-03-09T08:48:02.833 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e8067820 con 0x7f50e80a3950 2021-03-09T08:48:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.793+0000 7f50ee48f700 10 monclient(hunting): authenticate will time out at 848.283945s 2021-03-09T08:48:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50e80a5fd0 0x7f50e8141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.834 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50e80a5fd0 0x7f50e8141c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45772/0 (socket says 172.21.15.195:45772) 2021-03-09T08:48:02.835 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 -- 172.21.15.195:0/598798231 learned_addr learned my addr 172.21.15.195:0/598798231 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:02.835 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 10 monclient(hunting): get_auth_request con 0x7f50e80a5fd0 auth_method 0 2021-03-09T08:48:02.835 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.835 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.835 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ed48d700 1 --2- 172.21.15.195:0/598798231 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e80a3950 0x7f50e80681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.836 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50edc8e700 1 --2- 172.21.15.195:0/598798231 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50e81304f0 0x7f50e8142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:02.836 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ed48d700 10 monclient(hunting): get_auth_request con 0x7f50e80a3950 auth_method 0 2021-03-09T08:48:02.836 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ed48d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.836 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ed48d700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.837 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50edc8e700 10 monclient(hunting): get_auth_request con 0x7f50e81304f0 auth_method 0 2021-03-09T08:48:02.837 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50edc8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:02.837 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50edc8e700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:02.837 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.837 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.838 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.838 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50edc8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.838 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50edc8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.838 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50edc8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.839 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ed48d700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:02.839 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ed48d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:02.839 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ed48d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:02.839 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 10 monclient(hunting): handle_auth_done global_id 4117 payload 995 2021-03-09T08:48:02.839 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 -- 172.21.15.195:0/598798231 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e80a3950 msgr2=0x7f50e80681d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.840 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 --2- 172.21.15.195:0/598798231 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e80a3950 0x7f50e80681d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.840 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 -- 172.21.15.195:0/598798231 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50e81304f0 msgr2=0x7f50e8142110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.840 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 --2- 172.21.15.195:0/598798231 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50e81304f0 0x7f50e8142110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.840 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 10 monclient: _finish_hunting 0 2021-03-09T08:48:02.840 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 monclient: found mon.noname-b 2021-03-09T08:48:02.841 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:02.841 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.195:3300/0 2021-03-09T08:48:02.841 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 -- 172.21.15.195:0/598798231 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e80679e0 con 0x7f50e80a5fd0 2021-03-09T08:48:02.841 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ecc8c700 1 --2- 172.21.15.195:0/598798231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50e80a5fd0 0x7f50e8141c20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f50d800c740 tx=0x7f50d8007af0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:02.841 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 client.?.objecter ms_handle_connect 0x7f50e80a5fd0 2021-03-09T08:48:02.842 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:02.842 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 1 -- 172.21.15.195:0/598798231 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d80072e0 con 0x7f50e80a5fd0 2021-03-09T08:48:02.842 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:02.842 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:02.843 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 monclient: dump: 2021-03-09T08:48:02.843 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:02.843 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.843 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.843 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.844 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:02.844 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:02.844 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:02.844 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:02.844 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:02.845 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:02.845 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 monclient: _finish_auth 0 2021-03-09T08:48:02.845 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:02.845 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 1 -- 172.21.15.195:0/598798231 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d8007440 con 0x7f50e80a5fd0 2021-03-09T08:48:02.845 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ee48f700 5 monclient: authenticate success, global_id 4117 2021-03-09T08:48:02.846 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ee48f700 10 monclient: _renew_subs 2021-03-09T08:48:02.846 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ee48f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:02.846 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ee48f700 1 -- 172.21.15.195:0/598798231 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e8142600 con 0x7f50e80a5fd0 2021-03-09T08:48:02.846 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:02.847 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 1 -- 172.21.15.195:0/598798231 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d8005850 con 0x7f50e80a5fd0 2021-03-09T08:48:02.847 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:02.847 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:02.847 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ee48f700 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:02.847 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 monclient: dump: 2021-03-09T08:48:02.848 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:02.848 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:02.848 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.848 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:02.848 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:02.849 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:02.849 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:02.849 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:02.849 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:02.850 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:02.850 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ee48f700 10 monclient: _renew_subs 2021-03-09T08:48:02.850 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ee48f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:02.850 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ee48f700 1 -- 172.21.15.195:0/598798231 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e8142900 con 0x7f50e80a5fd0 2021-03-09T08:48:02.850 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50ee48f700 1 librados: init done 2021-03-09T08:48:02.851 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d77fe700 4 set_mon_vals no callback set 2021-03-09T08:48:02.851 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 1 -- 172.21.15.195:0/598798231 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f50d8010070 con 0x7f50e80a5fd0 2021-03-09T08:48:02.851 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 1 -- 172.21.15.195:0/598798231 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f50d801c3f0 con 0x7f50e80a5fd0 2021-03-09T08:48:02.851 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 10 client.4117.objecter ms_dispatch 0x7f50e8001550 osd_map(2..2 src has 1..2) v4 2021-03-09T08:48:02.851 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2021-03-09T08:48:02.852 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 3 client.4117.objecter handle_osd_map decoding full epoch 2 2021-03-09T08:48:02.852 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.794+0000 7f50d57fa700 20 client.4117.objecter dump_active .. 0 homeless 2021-03-09T08:48:02.852 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.795+0000 7f50ceffd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:02.852 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.795+0000 7f50ceffd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:02.853 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.795+0000 7f50ceffd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:02.853 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.795+0000 7f50ceffd700 1 -- 172.21.15.195:0/598798231 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c4004da0 con 0x7f50e80a5fd0 2021-03-09T08:48:02.853 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.797+0000 7f50d57fa700 1 -- 172.21.15.195:0/598798231 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f50d8013450 con 0x7f50e80a5fd0 2021-03-09T08:48:02.853 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.797+0000 7f50d57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:02.853 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.797+0000 7f50d57fa700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:02.877 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.875+0000 7f50ceffd700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bfcce70f-6a3c-4d76-9a45-145c7d5e468d", "id": 0}] 2021-03-09T08:48:02.878 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.875+0000 7f50ceffd700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bfcce70f-6a3c-4d76-9a45-145c7d5e468d", "id": 0}] 2021-03-09T08:48:02.878 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.875+0000 7f50ceffd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:02.878 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.875+0000 7f50ceffd700 1 -- 172.21.15.195:0/598798231 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bfcce70f-6a3c-4d76-9a45-145c7d5e468d", "id": 0} v 0) v1 -- 0x7f50c4005080 con 0x7f50e80a5fd0 2021-03-09T08:48:02.880 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.878+0000 7f50d57fa700 1 -- 172.21.15.195:0/598798231 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfcce70f-6a3c-4d76-9a45-145c7d5e468d", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f50d80135f0 con 0x7f50e80a5fd0 2021-03-09T08:48:02.880 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.878+0000 7f50d57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bfcce70f-6a3c-4d76-9a45-145c7d5e468d", "id": 0}] 2021-03-09T08:48:02.880 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.878+0000 7f50d57fa700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:02.881 INFO:teuthology.orchestra.run.smithi195.stdout:0 2021-03-09T08:48:02.882 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 10 librados: watch_flush enter 2021-03-09T08:48:02.882 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 10 librados: watch_flush exit 2021-03-09T08:48:02.883 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 20 client.4117.objecter shutdown clearing up homeless session... 2021-03-09T08:48:02.883 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 10 client.4117.objecter successfully canceled tick 2021-03-09T08:48:02.883 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 10 monclient: shutdown 2021-03-09T08:48:02.883 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:02.884 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 -- 172.21.15.195:0/598798231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50e80a5fd0 msgr2=0x7f50e8141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:02.884 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 --2- 172.21.15.195:0/598798231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50e80a5fd0 0x7f50e8141c20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f50d800c740 tx=0x7f50d8007af0).stop 2021-03-09T08:48:02.884 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 -- 172.21.15.195:0/598798231 shutdown_connections 2021-03-09T08:48:02.884 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 --2- 172.21.15.195:0/598798231 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f50e81304f0 0x7f50e8142110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.884 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 --2- 172.21.15.195:0/598798231 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f50e80a3950 0x7f50e80681d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.885 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 --2- 172.21.15.195:0/598798231 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f50e80a5fd0 0x7f50e8141c20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:02.885 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 -- 172.21.15.195:0/598798231 >> 172.21.15.195:0/598798231 conn(0x7f50e80642c0 msgr2=0x7f50e805c810 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:02.885 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 -- 172.21.15.195:0/598798231 shutdown_connections 2021-03-09T08:48:02.885 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 -- 172.21.15.195:0/598798231 wait complete. 2021-03-09T08:48:02.886 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:02.879+0000 7f50ceffd700 1 librados: shutdown 2021-03-09T08:48:02.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new ff4574a3-9580-4767-b99d-f2d4e488b465 1 2021-03-09T08:48:03.021 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.019+0000 7f4ec460c700 10 monclient: get_monmap_and_config 2021-03-09T08:48:03.021 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.019+0000 7f4ec460c700 10 monclient: build_initial_monmap 2021-03-09T08:48:03.022 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.019+0000 7f4ec460c700 10 monclient: monmap: 2021-03-09T08:48:03.022 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:03.022 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.023 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:48:03.019968+0000 2021-03-09T08:48:03.023 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:48:03.019968+0000 2021-03-09T08:48:03.023 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:03.023 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.023 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:03.024 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:03.024 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:03.024 INFO:teuthology.orchestra.run.smithi195.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:03.024 INFO:teuthology.orchestra.run.smithi195.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:03.025 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.025 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.019+0000 7f4ec460c700 1 Processor -- start 2021-03-09T08:48:03.025 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.019+0000 7f4ec460c700 1 -- start start 2021-03-09T08:48:03.025 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.019+0000 7f4ec460c700 10 monclient: init 2021-03-09T08:48:03.037 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.037 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2021-03-09T08:48:03.037 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4ebc0ada30 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:03.038 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4ebc0a6040 0x7f4ebc0a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.038 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 10 monclient(hunting): picked mon.noname-c con 0x7f4ebc0a6040 addr v2:172.21.15.195:3300/0 2021-03-09T08:48:03.038 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc138f80 0x7f4ebc139410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.038 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 10 monclient(hunting): picked mon.noname-b con 0x7f4ebc138f80 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:03.038 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0ade10 con 0x7f4ebc0ada30 2021-03-09T08:48:03.039 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.039 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ebc0a8dd0 con 0x7f4ebc0a6040 2021-03-09T08:48:03.039 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.039 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc0a8ac0 con 0x7f4ebc138f80 2021-03-09T08:48:03.040 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:03.040 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec460c700 10 monclient(hunting): authenticate will time out at 848.510612s 2021-03-09T08:48:03.040 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec1ba7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4ebc0a6040 0x7f4ebc0a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.040 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec1ba7700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f4ebc0a6040 0x7f4ebc0a6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45774/0 (socket says 172.21.15.195:45774) 2021-03-09T08:48:03.041 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec23a8700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f4ebc0ada30 0x7f4ebc0a5930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47986/0 (socket says 172.21.15.195:47986) 2021-03-09T08:48:03.041 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec1ba7700 1 -- 172.21.15.195:0/814822621 learned_addr learned my addr 172.21.15.195:0/814822621 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:03.041 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec1ba7700 10 monclient(hunting): get_auth_request con 0x7f4ebc0a6040 auth_method 0 2021-03-09T08:48:03.041 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec1ba7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.042 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec1ba7700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.042 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec1ba7700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.042 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec1ba7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.042 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec2ba9700 1 --2- 172.21.15.195:0/814822621 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc138f80 0x7f4ebc139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.042 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec1ba7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.043 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec2ba9700 10 monclient(hunting): get_auth_request con 0x7f4ebc138f80 auth_method 0 2021-03-09T08:48:03.043 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec2ba9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.043 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.020+0000 7f4ec2ba9700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.043 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec2ba9700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.044 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec2ba9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.044 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec2ba9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.044 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 1 -- 172.21.15.195:0/814822621 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608191713 0 0) 0x7f4eb00011d0 con 0x7f4ebc0ada30 2021-03-09T08:48:03.044 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.044 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient(hunting): my global_id is 4123 2021-03-09T08:48:03.045 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 1 -- 172.21.15.195:0/814822621 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea4003270 con 0x7f4ebc0ada30 2021-03-09T08:48:03.045 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 10 monclient(hunting): handle_auth_done global_id 4120 payload 450 2021-03-09T08:48:03.045 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 1 -- 172.21.15.195:0/814822621 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc138f80 msgr2=0x7f4ebc139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:48:03.045 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 1 --2- 172.21.15.195:0/814822621 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc138f80 0x7f4ebc139410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.046 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 1 -- 172.21.15.195:0/814822621 >> v1:172.21.15.195:6789/0 conn(0x7f4ebc0ada30 legacy=0x7f4ebc0a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.046 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 10 monclient: _finish_hunting 0 2021-03-09T08:48:03.046 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 1 monclient: found mon.noname-c 2021-03-09T08:48:03.046 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:03.047 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.047 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 1 -- 172.21.15.195:0/814822621 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc0a4480 con 0x7f4ebc0a6040 2021-03-09T08:48:03.047 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec1ba7700 1 --2- 172.21.15.195:0/814822621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 0x7f4ebc0a6be0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f4eac00ba20 tx=0x7f4eac006ee0).ready entity=mon.1 client_cookie=6ca699a1a8c7c4e0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:03.047 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 1 -- 172.21.15.195:0/814822621 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac00be60 con 0x7f4ebc0a6040 2021-03-09T08:48:03.048 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.048 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T08:48:03.048 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient: dump: 2021-03-09T08:48:03.048 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.048 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.049 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.049 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.049 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.049 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.050 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.050 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.050 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.050 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.051 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient: _finish_auth 0 2021-03-09T08:48:03.051 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:03.051 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 5 monclient: authenticate success, global_id 4120 2021-03-09T08:48:03.051 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:03.051 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 1 -- 172.21.15.195:0/814822621 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eac009840 con 0x7f4ebc0a6040 2021-03-09T08:48:03.052 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:03.052 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:03.052 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 4 set_mon_vals no callback set 2021-03-09T08:48:03.052 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 1 -- 172.21.15.195:0/814822621 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac009bc0 con 0x7f4ebc0a6040 2021-03-09T08:48:03.053 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.053 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:03.053 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec13a6700 10 monclient: dump: 2021-03-09T08:48:03.053 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.053 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.054 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.054 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.054 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.054 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.055 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.055 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.055 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.055 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.055 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 10 monclient: shutdown 2021-03-09T08:48:03.056 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:03.056 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 1 -- 172.21.15.195:0/814822621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 msgr2=0x7f4ebc0a6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.056 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 1 --2- 172.21.15.195:0/814822621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 0x7f4ebc0a6be0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f4eac00ba20 tx=0x7f4eac006ee0).stop 2021-03-09T08:48:03.056 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 1 -- 172.21.15.195:0/814822621 shutdown_connections 2021-03-09T08:48:03.057 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 1 --2- 172.21.15.195:0/814822621 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc138f80 0x7f4ebc139410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.057 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.021+0000 7f4ec460c700 1 --2- 172.21.15.195:0/814822621 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 0x7f4ebc0a6be0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.057 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 1 -- 172.21.15.195:0/814822621 >> 172.21.15.195:0/814822621 conn(0x7f4ebc0631d0 msgr2=0x7f4ebc0635b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:03.057 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 1 -- 172.21.15.195:0/814822621 shutdown_connections 2021-03-09T08:48:03.057 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 1 -- 172.21.15.195:0/814822621 wait complete. 2021-03-09T08:48:03.058 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 10 monclient: build_initial_monmap 2021-03-09T08:48:03.058 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 10 monclient: monmap: 2021-03-09T08:48:03.058 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:03.058 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:03.059 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2021-03-09T08:48:03.059 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2021-03-09T08:48:03.059 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:03.059 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2021-03-09T08:48:03.060 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:03.060 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:03.060 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:03.060 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.061 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 1 librados: starting msgr at 2021-03-09T08:48:03.061 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 1 librados: starting objecter 2021-03-09T08:48:03.061 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 1 Processor -- start 2021-03-09T08:48:03.061 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 1 -- start start 2021-03-09T08:48:03.062 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 1 librados: setting wanted keys 2021-03-09T08:48:03.062 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 1 librados: calling monclient init 2021-03-09T08:48:03.062 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.022+0000 7f4ec460c700 10 monclient: init 2021-03-09T08:48:03.062 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.062 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:48:03.063 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 0x7f4ebc0a3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.063 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient(hunting): picked mon.noname-b con 0x7f4ebc0a6040 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:03.063 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc0ada30 0x7f4ebc117390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.063 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient(hunting): picked mon.noname-c con 0x7f4ebc0ada30 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:03.064 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ebc138f80 0x7f4ebc114f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.064 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient(hunting): picked mon.noname-a con 0x7f4ebc138f80 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:03.064 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.064 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc115c60 con 0x7f4ebc138f80 2021-03-09T08:48:03.064 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.065 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc116ef0 con 0x7f4ebc0a6040 2021-03-09T08:48:03.065 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.065 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ebc116b70 con 0x7f4ebc0ada30 2021-03-09T08:48:03.065 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:03.066 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec460c700 10 monclient(hunting): authenticate will time out at 848.513490s 2021-03-09T08:48:03.066 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 0x7f4ebc0a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.066 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 0x7f4ebc0a3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45780/0 (socket says 172.21.15.195:45780) 2021-03-09T08:48:03.066 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 -- 172.21.15.195:0/1157224217 learned_addr learned my addr 172.21.15.195:0/1157224217 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:03.067 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 10 monclient(hunting): get_auth_request con 0x7f4ebc0a6040 auth_method 0 2021-03-09T08:48:03.067 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.067 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.067 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec1ba7700 1 --2- 172.21.15.195:0/1157224217 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc0ada30 0x7f4ebc117390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.068 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec1ba7700 10 monclient(hunting): get_auth_request con 0x7f4ebc0ada30 auth_method 0 2021-03-09T08:48:03.068 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec1ba7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.068 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec1ba7700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.068 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.069 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.069 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.069 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec2ba9700 1 --2- 172.21.15.195:0/1157224217 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ebc138f80 0x7f4ebc114f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.069 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec2ba9700 10 monclient(hunting): get_auth_request con 0x7f4ebc138f80 auth_method 0 2021-03-09T08:48:03.069 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec2ba9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.070 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec2ba9700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.070 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec1ba7700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.070 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec1ba7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.070 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec1ba7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.071 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec2ba9700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.071 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec2ba9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.071 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec2ba9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.071 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 10 monclient(hunting): handle_auth_done global_id 4126 payload 995 2021-03-09T08:48:03.072 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 -- 172.21.15.195:0/1157224217 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc0ada30 msgr2=0x7f4ebc117390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.072 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 --2- 172.21.15.195:0/1157224217 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc0ada30 0x7f4ebc117390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.072 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 -- 172.21.15.195:0/1157224217 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ebc138f80 msgr2=0x7f4ebc114f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.072 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 --2- 172.21.15.195:0/1157224217 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ebc138f80 0x7f4ebc114f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.072 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 10 monclient: _finish_hunting 0 2021-03-09T08:48:03.073 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 monclient: found mon.noname-b 2021-03-09T08:48:03.073 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:03.073 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.073 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 -- 172.21.15.195:0/1157224217 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc115580 con 0x7f4ebc0a6040 2021-03-09T08:48:03.074 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4ec23a8700 1 --2- 172.21.15.195:0/1157224217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 0x7f4ebc0a3030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4eb000c700 tx=0x7f4eb00028c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:03.074 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4eaa7fc700 10 client.?.objecter ms_handle_connect 0x7f4ebc0a6040 2021-03-09T08:48:03.074 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.023+0000 7f4eaa7fc700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:03.074 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 1 -- 172.21.15.195:0/1157224217 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb00072e0 con 0x7f4ebc0a6040 2021-03-09T08:48:03.075 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.075 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:03.075 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 10 monclient: dump: 2021-03-09T08:48:03.075 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.075 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.076 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.076 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.076 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.076 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.077 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.077 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.077 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.077 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.077 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 10 monclient: _finish_auth 0 2021-03-09T08:48:03.078 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:03.078 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 1 -- 172.21.15.195:0/1157224217 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb00054f0 con 0x7f4ebc0a6040 2021-03-09T08:48:03.078 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec460c700 5 monclient: authenticate success, global_id 4126 2021-03-09T08:48:03.078 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec460c700 10 monclient: _renew_subs 2021-03-09T08:48:03.079 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec460c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.079 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec460c700 1 -- 172.21.15.195:0/1157224217 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc146160 con 0x7f4ebc0a6040 2021-03-09T08:48:03.079 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:03.079 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 1 -- 172.21.15.195:0/1157224217 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0005870 con 0x7f4ebc0a6040 2021-03-09T08:48:03.080 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.080 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:03.080 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 10 monclient: dump: 2021-03-09T08:48:03.080 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.080 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.081 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.081 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.081 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.081 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.081 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.082 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.082 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.082 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.082 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec0ba5700 4 set_mon_vals no callback set 2021-03-09T08:48:03.083 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec460c700 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:03.083 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec460c700 10 monclient: _renew_subs 2021-03-09T08:48:03.083 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec460c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.083 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec460c700 1 -- 172.21.15.195:0/1157224217 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc1466a0 con 0x7f4ebc0a6040 2021-03-09T08:48:03.084 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4ec460c700 1 librados: init done 2021-03-09T08:48:03.084 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 1 -- 172.21.15.195:0/1157224217 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f4eb0010070 con 0x7f4ebc0a6040 2021-03-09T08:48:03.084 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 1 -- 172.21.15.195:0/1157224217 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7f4eb0013bf0 con 0x7f4ebc0a6040 2021-03-09T08:48:03.084 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 10 client.4126.objecter ms_dispatch 0x7f4ebc001550 osd_map(3..3 src has 1..3) v4 2021-03-09T08:48:03.085 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2021-03-09T08:48:03.085 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 3 client.4126.objecter handle_osd_map decoding full epoch 3 2021-03-09T08:48:03.085 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.024+0000 7f4eaa7fc700 20 client.4126.objecter dump_active .. 0 homeless 2021-03-09T08:48:03.085 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.025+0000 7f4ec460c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.085 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.025+0000 7f4ec460c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.086 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.025+0000 7f4ec460c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.086 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.025+0000 7f4ec460c700 1 -- 172.21.15.195:0/1157224217 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc116d30 con 0x7f4ebc0a6040 2021-03-09T08:48:03.086 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.027+0000 7f4eaa7fc700 1 -- 172.21.15.195:0/1157224217 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f4ebc0ade10 con 0x7f4ebc0a6040 2021-03-09T08:48:03.086 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.027+0000 7f4eaa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.087 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.027+0000 7f4eaa7fc700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:03.105 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.103+0000 7f4ec460c700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ff4574a3-9580-4767-b99d-f2d4e488b465", "id": 1}] 2021-03-09T08:48:03.105 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.103+0000 7f4ec460c700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ff4574a3-9580-4767-b99d-f2d4e488b465", "id": 1}] 2021-03-09T08:48:03.106 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.103+0000 7f4ec460c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.106 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.103+0000 7f4ec460c700 1 -- 172.21.15.195:0/1157224217 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ff4574a3-9580-4767-b99d-f2d4e488b465", "id": 1} v 0) v1 -- 0x7f4ebc139610 con 0x7f4ebc0a6040 2021-03-09T08:48:03.108 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.107+0000 7f4eaa7fc700 1 -- 172.21.15.195:0/1157224217 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ff4574a3-9580-4767-b99d-f2d4e488b465", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4eb0013450 con 0x7f4ebc0a6040 2021-03-09T08:48:03.109 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.107+0000 7f4eaa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ff4574a3-9580-4767-b99d-f2d4e488b465", "id": 1}] 2021-03-09T08:48:03.109 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.107+0000 7f4eaa7fc700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:03.109 INFO:teuthology.orchestra.run.smithi195.stdout:1 2021-03-09T08:48:03.111 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.107+0000 7f4ec460c700 10 librados: watch_flush enter 2021-03-09T08:48:03.111 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.107+0000 7f4ec460c700 10 librados: watch_flush exit 2021-03-09T08:48:03.111 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.107+0000 7f4ec460c700 20 client.4126.objecter shutdown clearing up homeless session... 2021-03-09T08:48:03.112 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.107+0000 7f4ec460c700 10 client.4126.objecter successfully canceled tick 2021-03-09T08:48:03.112 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 10 monclient: shutdown 2021-03-09T08:48:03.112 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:03.112 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 -- 172.21.15.195:0/1157224217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 msgr2=0x7f4ebc0a3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.112 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 --2- 172.21.15.195:0/1157224217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 0x7f4ebc0a3030 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4eb000c700 tx=0x7f4eb00028c0).stop 2021-03-09T08:48:03.113 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 -- 172.21.15.195:0/1157224217 shutdown_connections 2021-03-09T08:48:03.113 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 --2- 172.21.15.195:0/1157224217 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f4ebc138f80 0x7f4ebc114f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.113 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 --2- 172.21.15.195:0/1157224217 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f4ebc0a6040 0x7f4ebc0a3030 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.113 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 --2- 172.21.15.195:0/1157224217 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f4ebc0ada30 0x7f4ebc117390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.113 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 -- 172.21.15.195:0/1157224217 >> 172.21.15.195:0/1157224217 conn(0x7f4ebc0631d0 msgr2=0x7f4ebc0a5930 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:03.114 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 -- 172.21.15.195:0/1157224217 shutdown_connections 2021-03-09T08:48:03.114 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 -- 172.21.15.195:0/1157224217 wait complete. 2021-03-09T08:48:03.114 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.108+0000 7f4ec460c700 1 librados: shutdown 2021-03-09T08:48:03.121 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new eb8b7997-db54-48ba-8336-f8566c57204f 2 2021-03-09T08:48:03.252 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.250+0000 7fe678e2e700 10 monclient: get_monmap_and_config 2021-03-09T08:48:03.252 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.250+0000 7fe678e2e700 10 monclient: build_initial_monmap 2021-03-09T08:48:03.253 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.250+0000 7fe678e2e700 10 monclient: monmap: 2021-03-09T08:48:03.253 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:03.253 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.253 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:48:03.250985+0000 2021-03-09T08:48:03.253 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:48:03.250985+0000 2021-03-09T08:48:03.254 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:03.254 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.254 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:03.254 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:03.255 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:03.255 INFO:teuthology.orchestra.run.smithi195.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:03.255 INFO:teuthology.orchestra.run.smithi195.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:03.255 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.255 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.250+0000 7fe678e2e700 1 Processor -- start 2021-03-09T08:48:03.256 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.250+0000 7fe678e2e700 1 -- start start 2021-03-09T08:48:03.256 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.250+0000 7fe678e2e700 10 monclient: init 2021-03-09T08:48:03.265 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.266 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2021-03-09T08:48:03.266 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6740ada30 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:03.266 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740a6040 0x7fe6740a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.267 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 10 monclient(hunting): picked mon.noname-b con 0x7fe6740a6040 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:03.267 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe674138f80 addr v1:172.21.15.99:6789/0 2021-03-09T08:48:03.267 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6740ade10 con 0x7fe674138f80 2021-03-09T08:48:03.267 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6740a8dd0 con 0x7fe6740ada30 2021-03-09T08:48:03.268 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.268 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6740a8ac0 con 0x7fe6740a6040 2021-03-09T08:48:03.268 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:03.268 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe678e2e700 10 monclient(hunting): authenticate will time out at 848.741599s 2021-03-09T08:48:03.269 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe67259c700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7fe6740ada30 0x7fe6740a5930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:47998/0 (socket says 172.21.15.195:47998) 2021-03-09T08:48:03.269 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe67259c700 1 -- 172.21.15.195:0/1592650930 learned_addr learned my addr 172.21.15.195:0/1592650930 (peer_addr_for_me v1:172.21.15.195:0/0) 2021-03-09T08:48:03.269 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe671d9b700 1 --2- 172.21.15.195:0/1592650930 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740a6040 0x7fe6740a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.269 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe671d9b700 10 monclient(hunting): get_auth_request con 0x7fe6740a6040 auth_method 0 2021-03-09T08:48:03.270 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe671d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.270 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe671d9b700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.270 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe67159a700 1 -- 172.21.15.195:0/1592650930 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865701615 0 0) 0x7fe6740a8dd0 con 0x7fe6740ada30 2021-03-09T08:48:03.270 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe67159a700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.270 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.251+0000 7fe67159a700 10 monclient(hunting): my global_id is 4129 2021-03-09T08:48:03.271 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 1 -- 172.21.15.195:0/1592650930 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe658003270 con 0x7fe6740ada30 2021-03-09T08:48:03.271 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.271 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.271 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.272 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 1 -- 172.21.15.195:0/1592650930 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938428345 0 0) 0x7fe664001490 con 0x7fe674138f80 2021-03-09T08:48:03.272 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.272 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient(hunting): my global_id is 4122 2021-03-09T08:48:03.272 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 1 -- 172.21.15.195:0/1592650930 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6740a8dd0 con 0x7fe674138f80 2021-03-09T08:48:03.272 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 10 monclient(hunting): handle_auth_done global_id 4118 payload 450 2021-03-09T08:48:03.273 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 1 -- 172.21.15.195:0/1592650930 >> v1:172.21.15.195:6789/0 conn(0x7fe6740ada30 legacy=0x7fe6740a5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.273 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 1 -- 172.21.15.195:0/1592650930 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2513618941 0 0) 0x7fe658003270 con 0x7fe6740ada30 2021-03-09T08:48:03.273 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 1 -- 172.21.15.195:0/1592650930 >> v1:172.21.15.99:6789/0 conn(0x7fe674138f80 legacy=0x7fe6741393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.273 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 10 monclient: _finish_hunting 0 2021-03-09T08:48:03.274 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 1 monclient: found mon.noname-b 2021-03-09T08:48:03.274 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:03.274 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.99:3301/0 2021-03-09T08:48:03.274 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 1 -- 172.21.15.195:0/1592650930 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6740a4480 con 0x7fe6740a6040 2021-03-09T08:48:03.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2021-03-09T08:48:03.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe671d9b700 1 --2- 172.21.15.195:0/1592650930 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740a6040 0x7fe6740a6be0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe66800b9f0 tx=0x7fe668006e20).ready entity=mon.2 client_cookie=1768956e48b3171b server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:03.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 1 -- 172.21.15.195:0/1592650930 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668009630 con 0x7fe6740a6040 2021-03-09T08:48:03.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:03.276 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient: dump: 2021-03-09T08:48:03.276 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.276 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.276 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.277 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.277 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.277 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.277 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.278 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.278 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.278 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.278 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient: _finish_auth 0 2021-03-09T08:48:03.279 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:03.279 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 1 -- 172.21.15.195:0/1592650930 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe668009790 con 0x7fe6740a6040 2021-03-09T08:48:03.279 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe678e2e700 5 monclient: authenticate success, global_id 4118 2021-03-09T08:48:03.279 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe678e2e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:03.279 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:03.280 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe678e2e700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:03.280 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 1 -- 172.21.15.195:0/1592650930 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe668009b10 con 0x7fe6740a6040 2021-03-09T08:48:03.280 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe678e2e700 4 set_mon_vals no callback set 2021-03-09T08:48:03.280 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.281 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:48:03.281 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.252+0000 7fe67159a700 10 monclient: dump: 2021-03-09T08:48:03.281 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.281 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.281 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.282 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.282 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.282 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.282 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.283 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.283 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.283 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.283 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 10 monclient: shutdown 2021-03-09T08:48:03.284 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:03.284 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 1 -- 172.21.15.195:0/1592650930 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740a6040 msgr2=0x7fe6740a6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.284 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 1 --2- 172.21.15.195:0/1592650930 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740a6040 0x7fe6740a6be0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fe66800b9f0 tx=0x7fe668006e20).stop 2021-03-09T08:48:03.284 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 1 -- 172.21.15.195:0/1592650930 shutdown_connections 2021-03-09T08:48:03.284 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 1 --2- 172.21.15.195:0/1592650930 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740a6040 0x7fe6740a6be0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.285 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 1 -- 172.21.15.195:0/1592650930 >> 172.21.15.195:0/1592650930 conn(0x7fe6740631d0 msgr2=0x7fe6740635b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:03.285 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 1 -- 172.21.15.195:0/1592650930 shutdown_connections 2021-03-09T08:48:03.285 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 1 -- 172.21.15.195:0/1592650930 wait complete. 2021-03-09T08:48:03.285 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 10 monclient: build_initial_monmap 2021-03-09T08:48:03.286 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.253+0000 7fe678e2e700 10 monclient: monmap: 2021-03-09T08:48:03.286 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:03.286 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:03.286 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2021-03-09T08:48:03.286 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2021-03-09T08:48:03.287 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:03.287 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2021-03-09T08:48:03.287 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:03.287 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:03.288 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:03.288 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.288 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 librados: starting msgr at 2021-03-09T08:48:03.288 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 librados: starting objecter 2021-03-09T08:48:03.288 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 Processor -- start 2021-03-09T08:48:03.289 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 -- start start 2021-03-09T08:48:03.289 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 librados: setting wanted keys 2021-03-09T08:48:03.289 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 librados: calling monclient init 2021-03-09T08:48:03.289 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient: init 2021-03-09T08:48:03.290 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.290 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:48:03.290 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6740a6040 0x7fe674135e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.290 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient(hunting): picked mon.noname-a con 0x7fe6740a6040 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:03.290 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740ada30 0x7fe674149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.291 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient(hunting): picked mon.noname-c con 0x7fe6740ada30 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:03.291 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe674138f80 0x7fe67414a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.291 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient(hunting): picked mon.noname-b con 0x7fe674138f80 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:03.291 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.292 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe674136370 con 0x7fe6740a6040 2021-03-09T08:48:03.292 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.292 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6741495c0 con 0x7fe674138f80 2021-03-09T08:48:03.292 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.293 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe674135700 con 0x7fe6740ada30 2021-03-09T08:48:03.293 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:03.293 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe678e2e700 10 monclient(hunting): authenticate will time out at 848.744723s 2021-03-09T08:48:03.293 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe672d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe674138f80 0x7fe67414a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.293 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe672d9d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe674138f80 0x7fe67414a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45796/0 (socket says 172.21.15.195:45796) 2021-03-09T08:48:03.294 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe672d9d700 1 -- 172.21.15.195:0/609885627 learned_addr learned my addr 172.21.15.195:0/609885627 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:03.294 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe672d9d700 10 monclient(hunting): get_auth_request con 0x7fe674138f80 auth_method 0 2021-03-09T08:48:03.294 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe672d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.294 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe672d9d700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.295 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe672d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.295 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe672d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.295 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe67259c700 1 --2- 172.21.15.195:0/609885627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6740a6040 0x7fe674135e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.295 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe672d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.296 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe67259c700 10 monclient(hunting): get_auth_request con 0x7fe6740a6040 auth_method 0 2021-03-09T08:48:03.296 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe67259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.296 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.254+0000 7fe67259c700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.297 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe671d9b700 1 --2- 172.21.15.195:0/609885627 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740ada30 0x7fe674149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.297 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe671d9b700 10 monclient(hunting): get_auth_request con 0x7fe6740ada30 auth_method 0 2021-03-09T08:48:03.297 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe671d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.297 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe671d9b700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.297 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe67259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.298 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe67259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.298 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe67259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.298 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 10 monclient(hunting): handle_auth_done global_id 4132 payload 995 2021-03-09T08:48:03.298 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 1 -- 172.21.15.195:0/609885627 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740ada30 msgr2=0x7fe674149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.299 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 1 --2- 172.21.15.195:0/609885627 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740ada30 0x7fe674149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.304 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 1 -- 172.21.15.195:0/609885627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6740a6040 msgr2=0x7fe674135e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.304 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 1 --2- 172.21.15.195:0/609885627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6740a6040 0x7fe674135e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.305 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 10 monclient: _finish_hunting 0 2021-03-09T08:48:03.305 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 1 monclient: found mon.noname-b 2021-03-09T08:48:03.305 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:03.305 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.305 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 1 -- 172.21.15.195:0/609885627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6740a2890 con 0x7fe674138f80 2021-03-09T08:48:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe672d9d700 1 --2- 172.21.15.195:0/609885627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe674138f80 0x7fe67414a160 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe66400ca60 tx=0x7fe664007c80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 client.?.objecter ms_handle_connect 0x7fe674138f80 2021-03-09T08:48:03.306 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:03.307 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 1 -- 172.21.15.195:0/609885627 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66400ce20 con 0x7fe674138f80 2021-03-09T08:48:03.307 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.307 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:03.307 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 monclient: dump: 2021-03-09T08:48:03.307 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.308 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.308 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.308 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.308 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.309 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.309 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.309 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.309 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.310 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.310 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 monclient: _finish_auth 0 2021-03-09T08:48:03.310 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:03.310 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 1 -- 172.21.15.195:0/609885627 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66400a7a0 con 0x7fe674138f80 2021-03-09T08:48:03.311 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe678e2e700 5 monclient: authenticate success, global_id 4132 2021-03-09T08:48:03.311 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe678e2e700 10 monclient: _renew_subs 2021-03-09T08:48:03.311 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe678e2e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.311 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe678e2e700 1 -- 172.21.15.195:0/609885627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6740a2b40 con 0x7fe674138f80 2021-03-09T08:48:03.312 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:03.312 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 1 -- 172.21.15.195:0/609885627 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66400ab40 con 0x7fe674138f80 2021-03-09T08:48:03.312 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.312 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:03.312 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe670d99700 4 set_mon_vals no callback set 2021-03-09T08:48:03.313 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 monclient: dump: 2021-03-09T08:48:03.313 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.313 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.313 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.313 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.314 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.314 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.314 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.314 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.315 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.315 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.315 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe678e2e700 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:03.315 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe678e2e700 10 monclient: _renew_subs 2021-03-09T08:48:03.315 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe678e2e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.316 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe678e2e700 1 -- 172.21.15.195:0/609885627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6740a3000 con 0x7fe674138f80 2021-03-09T08:48:03.316 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe678e2e700 1 librados: init done 2021-03-09T08:48:03.316 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 1 -- 172.21.15.195:0/609885627 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7fe664010070 con 0x7fe674138f80 2021-03-09T08:48:03.316 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 1 -- 172.21.15.195:0/609885627 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7fe66401bca0 con 0x7fe674138f80 2021-03-09T08:48:03.316 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 10 client.4132.objecter ms_dispatch 0x7fe674001550 osd_map(4..4 src has 1..4) v4 2021-03-09T08:48:03.317 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2021-03-09T08:48:03.317 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 3 client.4132.objecter handle_osd_map decoding full epoch 4 2021-03-09T08:48:03.317 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.255+0000 7fe6627fc700 20 client.4132.objecter dump_active .. 0 homeless 2021-03-09T08:48:03.317 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.256+0000 7fe678e2e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.317 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.256+0000 7fe678e2e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.318 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.256+0000 7fe678e2e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.318 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.256+0000 7fe678e2e700 1 -- 172.21.15.195:0/609885627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6740a3cb0 con 0x7fe674138f80 2021-03-09T08:48:03.318 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.258+0000 7fe6627fc700 1 -- 172.21.15.195:0/609885627 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7fe6740ade10 con 0x7fe674138f80 2021-03-09T08:48:03.318 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.258+0000 7fe6627fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.318 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.258+0000 7fe6627fc700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:03.372 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.370+0000 7fe678e2e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eb8b7997-db54-48ba-8336-f8566c57204f", "id": 2}] 2021-03-09T08:48:03.372 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.370+0000 7fe678e2e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eb8b7997-db54-48ba-8336-f8566c57204f", "id": 2}] 2021-03-09T08:48:03.373 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.370+0000 7fe678e2e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.373 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.370+0000 7fe678e2e700 1 -- 172.21.15.195:0/609885627 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "eb8b7997-db54-48ba-8336-f8566c57204f", "id": 2} v 0) v1 -- 0x7fe6740a32b0 con 0x7fe674138f80 2021-03-09T08:48:03.375 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe6627fc700 1 -- 172.21.15.195:0/609885627 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb8b7997-db54-48ba-8336-f8566c57204f", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe66401b400 con 0x7fe674138f80 2021-03-09T08:48:03.375 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe6627fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eb8b7997-db54-48ba-8336-f8566c57204f", "id": 2}] 2021-03-09T08:48:03.376 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe6627fc700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:03.376 INFO:teuthology.orchestra.run.smithi195.stdout:2 2021-03-09T08:48:03.378 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe678e2e700 10 librados: watch_flush enter 2021-03-09T08:48:03.378 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe678e2e700 10 librados: watch_flush exit 2021-03-09T08:48:03.378 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe678e2e700 20 client.4132.objecter shutdown clearing up homeless session... 2021-03-09T08:48:03.379 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe678e2e700 10 client.4132.objecter successfully canceled tick 2021-03-09T08:48:03.379 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe678e2e700 10 monclient: shutdown 2021-03-09T08:48:03.379 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe678e2e700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:03.379 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.374+0000 7fe678e2e700 1 -- 172.21.15.195:0/609885627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe674138f80 msgr2=0x7fe67414a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.379 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.375+0000 7fe678e2e700 1 --2- 172.21.15.195:0/609885627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe674138f80 0x7fe67414a160 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe66400ca60 tx=0x7fe664007c80).stop 2021-03-09T08:48:03.380 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.375+0000 7fe678e2e700 1 -- 172.21.15.195:0/609885627 shutdown_connections 2021-03-09T08:48:03.380 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.375+0000 7fe678e2e700 1 --2- 172.21.15.195:0/609885627 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fe674138f80 0x7fe67414a160 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.380 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.375+0000 7fe678e2e700 1 --2- 172.21.15.195:0/609885627 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fe6740a6040 0x7fe674135e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.380 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.375+0000 7fe678e2e700 1 --2- 172.21.15.195:0/609885627 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fe6740ada30 0x7fe674149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.380 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.375+0000 7fe678e2e700 1 -- 172.21.15.195:0/609885627 >> 172.21.15.195:0/609885627 conn(0x7fe6740631d0 msgr2=0x7fe6740635b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:03.381 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.375+0000 7fe678e2e700 1 -- 172.21.15.195:0/609885627 shutdown_connections 2021-03-09T08:48:03.381 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.375+0000 7fe678e2e700 1 -- 172.21.15.195:0/609885627 wait complete. 2021-03-09T08:48:03.381 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.375+0000 7fe678e2e700 1 librados: shutdown 2021-03-09T08:48:03.388 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new dc1a81b3-b2c9-4c37-9429-6d131c9e791c 3 2021-03-09T08:48:03.518 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.515+0000 7f5d53fff700 10 monclient: get_monmap_and_config 2021-03-09T08:48:03.518 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.515+0000 7f5d53fff700 10 monclient: build_initial_monmap 2021-03-09T08:48:03.518 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.515+0000 7f5d53fff700 10 monclient: monmap: 2021-03-09T08:48:03.519 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:03.519 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.519 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:48:03.516571+0000 2021-03-09T08:48:03.519 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:48:03.516571+0000 2021-03-09T08:48:03.520 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:03.520 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.520 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:03.520 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:03.521 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:03.521 INFO:teuthology.orchestra.run.smithi195.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:03.521 INFO:teuthology.orchestra.run.smithi195.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:03.521 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.521 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.516+0000 7f5d53fff700 1 Processor -- start 2021-03-09T08:48:03.522 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.516+0000 7f5d53fff700 1 -- start start 2021-03-09T08:48:03.522 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.516+0000 7f5d53fff700 10 monclient: init 2021-03-09T08:48:03.537 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.537 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2021-03-09T08:48:03.538 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d5406a860 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:03.538 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d54068ad0 0x7f5d54061b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.538 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 10 monclient(hunting): picked mon.noname-b con 0x7f5d54068ad0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:03.538 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5d54062060 addr v1:172.21.15.99:6789/0 2021-03-09T08:48:03.539 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d540a6840 con 0x7f5d54062060 2021-03-09T08:48:03.539 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5406ac40 con 0x7f5d5406a860 2021-03-09T08:48:03.539 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.539 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54062ee0 con 0x7f5d54068ad0 2021-03-09T08:48:03.540 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:03.540 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 10 monclient(hunting): authenticate will time out at 849.007146s 2021-03-09T08:48:03.540 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d52ffd700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f5d5406a860 0x7f5d540683c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48010/0 (socket says 172.21.15.195:48010) 2021-03-09T08:48:03.540 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d52ffd700 1 -- 172.21.15.195:0/1608942931 learned_addr learned my addr 172.21.15.195:0/1608942931 (peer_addr_for_me v1:172.21.15.195:0/0) 2021-03-09T08:48:03.540 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d527fc700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d54068ad0 0x7f5d54061b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.541 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d527fc700 10 monclient(hunting): get_auth_request con 0x7f5d54068ad0 auth_method 0 2021-03-09T08:48:03.541 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d527fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.541 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 <== mon.1 v1:172.21.15.195:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701510832 0 0) 0x7f5d3c001110 con 0x7f5d5406a860 2021-03-09T08:48:03.541 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d527fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.542 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.542 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): my global_id is 4135 2021-03-09T08:48:03.542 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 --> v1:172.21.15.195:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d40003270 con 0x7f5d5406a860 2021-03-09T08:48:03.542 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d527fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.548 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d527fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.548 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d527fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.548 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348559015 0 0) 0x7f5d540a6840 con 0x7f5d54062060 2021-03-09T08:48:03.548 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.549 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): my global_id is 4128 2021-03-09T08:48:03.549 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 --> v1:172.21.15.99:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d3c001110 con 0x7f5d54062060 2021-03-09T08:48:03.549 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 <== mon.1 v1:172.21.15.195:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 878177291 0 0) 0x7f5d40003270 con 0x7f5d5406a860 2021-03-09T08:48:03.549 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d54068ad0 msgr2=0x7f5d54061b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:48:03.550 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d54068ad0 0x7f5d54061b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.550 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 >> v1:172.21.15.99:6789/0 conn(0x7f5d54062060 legacy=0x7f5d540624f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.550 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient: _finish_hunting 0 2021-03-09T08:48:03.550 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 monclient: found mon.noname-c-legacy 2021-03-09T08:48:03.551 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:03.551 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.195:6789/0 2021-03-09T08:48:03.551 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 --> v1:172.21.15.195:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d540a1be0 con 0x7f5d5406a860 2021-03-09T08:48:03.551 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient: _finish_auth 0 2021-03-09T08:48:03.551 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:03.552 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 <== mon.1 v1:172.21.15.195:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1912569104 0 0) 0x7f5d3c0016b0 con 0x7f5d5406a860 2021-03-09T08:48:03.552 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 5 monclient: authenticate success, global_id 4135 2021-03-09T08:48:03.552 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:03.553 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.554 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2021-03-09T08:48:03.554 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient: dump: 2021-03-09T08:48:03.554 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.554 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.555 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.555 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.555 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.555 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.555 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.556 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.556 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.556 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.556 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] but i'm connected to v1:172.21.15.195:6789/0, reconnecting 2021-03-09T08:48:03.557 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.557 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 >> v1:172.21.15.195:6789/0 conn(0x7f5d5406a860 legacy=0x7f5d540683c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.557 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:48:03.557 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d54068ad0 0x7f5d400038f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.557 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): picked mon.c con 0x7f5d54068ad0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:03.558 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d40003de0 0x7f5d40004510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.558 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): picked mon.a con 0x7f5d40003de0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:03.558 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d40004a00 0x7f5d40004e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.558 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): picked mon.b con 0x7f5d40004a00 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:03.559 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.559 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54062ee0 con 0x7f5d40003de0 2021-03-09T08:48:03.559 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.559 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4000b760 con 0x7f5d40004a00 2021-03-09T08:48:03.560 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.560 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4000b8a0 con 0x7f5d54068ad0 2021-03-09T08:48:03.560 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d51ffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:03.560 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.517+0000 7f5d53fff700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:03.561 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d40004a00 0x7f5d40004e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.561 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient(hunting): get_auth_request con 0x7f5d40004a00 auth_method 0 2021-03-09T08:48:03.561 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.561 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.561 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d52ffd700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d54068ad0 0x7f5d400038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.562 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d52ffd700 10 monclient(hunting): get_auth_request con 0x7f5d54068ad0 auth_method 0 2021-03-09T08:48:03.562 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.562 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d527fc700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d40003de0 0x7f5d40004510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.562 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d52ffd700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.563 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.563 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.563 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.563 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d527fc700 10 monclient(hunting): get_auth_request con 0x7f5d40003de0 auth_method 0 2021-03-09T08:48:03.563 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d527fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.564 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d527fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.564 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d52ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.564 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d52ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.564 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d52ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.565 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d527fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.565 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d527fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.565 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d527fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.565 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient(hunting): handle_auth_done global_id 4135 payload 450 2021-03-09T08:48:03.565 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 1 -- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d54068ad0 msgr2=0x7f5d400038f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:48:03.566 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d54068ad0 0x7f5d400038f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.566 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 1 -- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d40003de0 msgr2=0x7f5d40004510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:48:03.566 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d40003de0 0x7f5d40004510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.566 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient: _finish_hunting 0 2021-03-09T08:48:03.567 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 1 monclient: found mon.b 2021-03-09T08:48:03.567 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:03.567 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.567 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 1 -- 172.21.15.195:0/1608942931 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d40003270 con 0x7f5d40004a00 2021-03-09T08:48:03.568 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 10 monclient: _finish_auth 0 2021-03-09T08:48:03.568 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:03.568 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d40004a00 0x7f5d40004e80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5d44002040 tx=0x7f5d44007f40).ready entity=mon.1 client_cookie=6b3d3d685621eff4 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:03.568 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4400cb40 con 0x7f5d40004a00 2021-03-09T08:48:03.568 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.569 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:03.569 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 10 monclient: dump: 2021-03-09T08:48:03.569 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.569 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.570 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.570 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.570 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.570 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.570 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.571 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.571 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.571 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.571 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d53fff700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4400cca0 con 0x7f5d40004a00 2021-03-09T08:48:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d53fff700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d53fff700 4 set_mon_vals no callback set 2021-03-09T08:48:03.572 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 1 -- 172.21.15.195:0/1608942931 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44006ab0 con 0x7f5d40004a00 2021-03-09T08:48:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d51ffb700 10 monclient: dump: 2021-03-09T08:48:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.573 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.574 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.575 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.575 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.575 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.575 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d53fff700 10 monclient: shutdown 2021-03-09T08:48:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d53fff700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d53fff700 1 -- 172.21.15.195:0/1608942931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d40004a00 msgr2=0x7f5d40004e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d53fff700 1 --2- 172.21.15.195:0/1608942931 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d40004a00 0x7f5d40004e80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f5d44002040 tx=0x7f5d44007f40).stop 2021-03-09T08:48:03.576 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.518+0000 7f5d537fe700 1 -- 172.21.15.195:0/1608942931 reap_dead start 2021-03-09T08:48:03.577 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.519+0000 7f5d53fff700 1 -- 172.21.15.195:0/1608942931 shutdown_connections 2021-03-09T08:48:03.577 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.519+0000 7f5d53fff700 1 -- 172.21.15.195:0/1608942931 >> 172.21.15.195:0/1608942931 conn(0x7f5d540650e0 msgr2=0x7f5d54069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:03.577 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.519+0000 7f5d53fff700 1 -- 172.21.15.195:0/1608942931 shutdown_connections 2021-03-09T08:48:03.577 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.519+0000 7f5d53fff700 1 -- 172.21.15.195:0/1608942931 wait complete. 2021-03-09T08:48:03.578 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.519+0000 7f5d53fff700 10 monclient: build_initial_monmap 2021-03-09T08:48:03.578 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.519+0000 7f5d53fff700 10 monclient: monmap: 2021-03-09T08:48:03.578 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:03.578 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:03.579 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2021-03-09T08:48:03.579 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2021-03-09T08:48:03.579 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:03.579 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2021-03-09T08:48:03.579 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:03.580 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:03.580 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:03.580 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.580 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 librados: starting msgr at 2021-03-09T08:48:03.580 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 librados: starting objecter 2021-03-09T08:48:03.581 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 Processor -- start 2021-03-09T08:48:03.581 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 -- start start 2021-03-09T08:48:03.581 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 librados: setting wanted keys 2021-03-09T08:48:03.581 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 librados: calling monclient init 2021-03-09T08:48:03.582 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient: init 2021-03-09T08:48:03.582 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.582 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:48:03.582 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d540a3c30 0x7f5d5411bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.582 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient(hunting): picked mon.noname-c con 0x7f5d540a3c30 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:03.583 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d54145b80 0x7f5d5413c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.583 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient(hunting): picked mon.noname-a con 0x7f5d54145b80 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:03.583 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d5413ca60 0x7f5d54149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.583 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient(hunting): picked mon.noname-b con 0x7f5d5413ca60 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:03.584 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.584 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5411a610 con 0x7f5d54145b80 2021-03-09T08:48:03.584 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.584 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54118370 con 0x7f5d5413ca60 2021-03-09T08:48:03.584 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.585 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54149260 con 0x7f5d540a3c30 2021-03-09T08:48:03.585 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:03.585 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d53fff700 10 monclient(hunting): authenticate will time out at 849.010744s 2021-03-09T08:48:03.585 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d537fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d5413ca60 0x7f5d54149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.586 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d537fe700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d5413ca60 0x7f5d54149700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45814/0 (socket says 172.21.15.195:45814) 2021-03-09T08:48:03.586 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d537fe700 1 -- 172.21.15.195:0/2241902233 learned_addr learned my addr 172.21.15.195:0/2241902233 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:03.586 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d537fe700 10 monclient(hunting): get_auth_request con 0x7f5d5413ca60 auth_method 0 2021-03-09T08:48:03.586 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.586 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d537fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.587 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d52ffd700 1 --2- 172.21.15.195:0/2241902233 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d540a3c30 0x7f5d5411bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.587 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d527fc700 1 --2- 172.21.15.195:0/2241902233 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d54145b80 0x7f5d5413c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.587 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d52ffd700 10 monclient(hunting): get_auth_request con 0x7f5d540a3c30 auth_method 0 2021-03-09T08:48:03.587 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d52ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.588 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d52ffd700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.588 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d527fc700 10 monclient(hunting): get_auth_request con 0x7f5d54145b80 auth_method 0 2021-03-09T08:48:03.588 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d527fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.588 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d527fc700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.588 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.589 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.589 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.520+0000 7f5d537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.589 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d527fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.589 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d527fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.590 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d527fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.590 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d52ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.590 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d52ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.590 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d52ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.590 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 10 monclient(hunting): handle_auth_done global_id 4138 payload 995 2021-03-09T08:48:03.591 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 1 -- 172.21.15.195:0/2241902233 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d540a3c30 msgr2=0x7f5d5411bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.591 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 1 --2- 172.21.15.195:0/2241902233 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d540a3c30 0x7f5d5411bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.591 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 1 -- 172.21.15.195:0/2241902233 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d54145b80 msgr2=0x7f5d5413c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.591 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 1 --2- 172.21.15.195:0/2241902233 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d54145b80 0x7f5d5413c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 10 monclient: _finish_hunting 0 2021-03-09T08:48:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 1 monclient: found mon.noname-b 2021-03-09T08:48:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.592 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 1 -- 172.21.15.195:0/2241902233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5411c030 con 0x7f5d5413ca60 2021-03-09T08:48:03.593 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d537fe700 1 --2- 172.21.15.195:0/2241902233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d5413ca60 0x7f5d54149700 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5d40005470 tx=0x7f5d44016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:03.593 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 client.?.objecter ms_handle_connect 0x7f5d5413ca60 2021-03-09T08:48:03.593 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:03.593 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 1 -- 172.21.15.195:0/2241902233 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44006ab0 con 0x7f5d5413ca60 2021-03-09T08:48:03.594 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.594 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:03.594 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 monclient: dump: 2021-03-09T08:48:03.594 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.594 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.595 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.595 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.595 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.595 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.596 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.596 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.596 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.596 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.596 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 monclient: _finish_auth 0 2021-03-09T08:48:03.597 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:03.597 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d53fff700 5 monclient: authenticate success, global_id 4138 2021-03-09T08:48:03.597 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 1 -- 172.21.15.195:0/2241902233 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d440072c0 con 0x7f5d5413ca60 2021-03-09T08:48:03.597 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d53fff700 10 monclient: _renew_subs 2021-03-09T08:48:03.598 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d53fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.598 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d53fff700 1 -- 172.21.15.195:0/2241902233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d540a4010 con 0x7f5d5413ca60 2021-03-09T08:48:03.598 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:03.598 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 1 -- 172.21.15.195:0/2241902233 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44007640 con 0x7f5d5413ca60 2021-03-09T08:48:03.598 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.599 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:03.599 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d3affd700 10 monclient: dump: 2021-03-09T08:48:03.599 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.599 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.600 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.600 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.600 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.600 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.600 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.601 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.601 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.601 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.601 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d53fff700 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:03.601 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d517fa700 4 set_mon_vals no callback set 2021-03-09T08:48:03.602 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d53fff700 10 monclient: _renew_subs 2021-03-09T08:48:03.602 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d53fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.602 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d53fff700 1 -- 172.21.15.195:0/2241902233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d54149db0 con 0x7f5d5413ca60 2021-03-09T08:48:03.602 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.521+0000 7f5d53fff700 1 librados: init done 2021-03-09T08:48:03.603 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d3affd700 1 -- 172.21.15.195:0/2241902233 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f5d44025030 con 0x7f5d5413ca60 2021-03-09T08:48:03.603 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d3affd700 1 -- 172.21.15.195:0/2241902233 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f5d44028900 con 0x7f5d5413ca60 2021-03-09T08:48:03.603 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d3affd700 10 client.4138.objecter ms_dispatch 0x7f5d54001550 osd_map(5..5 src has 1..5) v4 2021-03-09T08:48:03.603 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d3affd700 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2021-03-09T08:48:03.604 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d3affd700 3 client.4138.objecter handle_osd_map decoding full epoch 5 2021-03-09T08:48:03.604 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d3affd700 20 client.4138.objecter dump_active .. 0 homeless 2021-03-09T08:48:03.604 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d53fff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.604 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d53fff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.604 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d53fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.605 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.522+0000 7f5d53fff700 1 -- 172.21.15.195:0/2241902233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d54149420 con 0x7f5d5413ca60 2021-03-09T08:48:03.605 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.524+0000 7f5d3affd700 1 -- 172.21.15.195:0/2241902233 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f5d4402d060 con 0x7f5d5413ca60 2021-03-09T08:48:03.605 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.524+0000 7f5d3affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.605 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.524+0000 7f5d3affd700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:03.607 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.602+0000 7f5d53fff700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dc1a81b3-b2c9-4c37-9429-6d131c9e791c", "id": 3}] 2021-03-09T08:48:03.607 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.602+0000 7f5d53fff700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dc1a81b3-b2c9-4c37-9429-6d131c9e791c", "id": 3}] 2021-03-09T08:48:03.607 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.602+0000 7f5d53fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.607 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.602+0000 7f5d53fff700 1 -- 172.21.15.195:0/2241902233 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dc1a81b3-b2c9-4c37-9429-6d131c9e791c", "id": 3} v 0) v1 -- 0x7f5d540589f0 con 0x7f5d5413ca60 2021-03-09T08:48:03.608 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.605+0000 7f5d3affd700 1 -- 172.21.15.195:0/2241902233 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dc1a81b3-b2c9-4c37-9429-6d131c9e791c", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5d44023070 con 0x7f5d5413ca60 2021-03-09T08:48:03.608 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.605+0000 7f5d3affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dc1a81b3-b2c9-4c37-9429-6d131c9e791c", "id": 3}] 2021-03-09T08:48:03.608 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.605+0000 7f5d3affd700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:03.609 INFO:teuthology.orchestra.run.smithi195.stdout:3 2021-03-09T08:48:03.611 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 10 librados: watch_flush enter 2021-03-09T08:48:03.611 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 10 librados: watch_flush exit 2021-03-09T08:48:03.611 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 20 client.4138.objecter shutdown clearing up homeless session... 2021-03-09T08:48:03.611 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 10 client.4138.objecter successfully canceled tick 2021-03-09T08:48:03.612 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 10 monclient: shutdown 2021-03-09T08:48:03.612 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:03.612 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 -- 172.21.15.195:0/2241902233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d5413ca60 msgr2=0x7f5d54149700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.612 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 --2- 172.21.15.195:0/2241902233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d5413ca60 0x7f5d54149700 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f5d40005470 tx=0x7f5d44016000).stop 2021-03-09T08:48:03.613 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 -- 172.21.15.195:0/2241902233 shutdown_connections 2021-03-09T08:48:03.613 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 --2- 172.21.15.195:0/2241902233 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f5d5413ca60 0x7f5d54149700 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.613 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 --2- 172.21.15.195:0/2241902233 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f5d540a3c30 0x7f5d5411bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.613 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 --2- 172.21.15.195:0/2241902233 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f5d54145b80 0x7f5d5413c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.614 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 -- 172.21.15.195:0/2241902233 >> 172.21.15.195:0/2241902233 conn(0x7f5d540650e0 msgr2=0x7f5d540a1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:03.614 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 -- 172.21.15.195:0/2241902233 shutdown_connections 2021-03-09T08:48:03.614 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 -- 172.21.15.195:0/2241902233 wait complete. 2021-03-09T08:48:03.614 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.606+0000 7f5d53fff700 1 librados: shutdown 2021-03-09T08:48:03.619 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 802be6f1-4e15-4e2d-8720-0a3009f8d02e 4 2021-03-09T08:48:03.748 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.746+0000 7f738e80c700 10 monclient: get_monmap_and_config 2021-03-09T08:48:03.749 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.746+0000 7f738e80c700 10 monclient: build_initial_monmap 2021-03-09T08:48:03.749 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.746+0000 7f738e80c700 10 monclient: monmap: 2021-03-09T08:48:03.749 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:03.749 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.750 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:48:03.747496+0000 2021-03-09T08:48:03.750 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:48:03.747496+0000 2021-03-09T08:48:03.750 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:03.750 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.751 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:03.751 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:03.751 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:03.751 INFO:teuthology.orchestra.run.smithi195.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:03.752 INFO:teuthology.orchestra.run.smithi195.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:03.752 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.765 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.747+0000 7f738e80c700 1 Processor -- start 2021-03-09T08:48:03.765 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.747+0000 7f738e80c700 1 -- start start 2021-03-09T08:48:03.765 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.747+0000 7f738e80c700 10 monclient: init 2021-03-09T08:48:03.765 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.766 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2021-03-09T08:48:03.766 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7388069f90 addr v1:172.21.15.99:6789/0 2021-03-09T08:48:03.766 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7388136050 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:03.766 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f73881364b0 0x7f738813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.767 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 10 monclient(hunting): picked mon.noname-c con 0x7f73881364b0 addr v2:172.21.15.195:3300/0 2021-03-09T08:48:03.767 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73880a5460 con 0x7f7388069f90 2021-03-09T08:48:03.767 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73880a4fc0 con 0x7f7388136050 2021-03-09T08:48:03.768 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.768 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73880a0960 con 0x7f73881364b0 2021-03-09T08:48:03.768 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:03.769 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 10 monclient(hunting): authenticate will time out at 849.238133s 2021-03-09T08:48:03.769 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f73881364b0 0x7f738813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.769 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f73877fe700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f7388136050 0x7f7388139fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48028/0 (socket says 172.21.15.195:48028) 2021-03-09T08:48:03.769 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f73877fe700 1 -- 172.21.15.195:0/3013550864 learned_addr learned my addr 172.21.15.195:0/3013550864 (peer_addr_for_me v1:172.21.15.195:0/0) 2021-03-09T08:48:03.769 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 10 monclient(hunting): get_auth_request con 0x7f73881364b0 auth_method 0 2021-03-09T08:48:03.770 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.770 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.770 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.771 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.771 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.771 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 10 monclient(hunting): handle_auth_done global_id 4141 payload 450 2021-03-09T08:48:03.771 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 1 -- 172.21.15.195:0/3013550864 >> v1:172.21.15.195:6789/0 conn(0x7f7388136050 legacy=0x7f7388139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.772 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 1 -- 172.21.15.195:0/3013550864 >> v1:172.21.15.99:6789/0 conn(0x7f7388069f90 legacy=0x7f73880a4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.772 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 10 monclient: _finish_hunting 0 2021-03-09T08:48:03.772 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 1 monclient: found mon.noname-c 2021-03-09T08:48:03.772 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:03.773 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.773 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 1 -- 172.21.15.195:0/3013550864 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73880a0aa0 con 0x7f73881364b0 2021-03-09T08:48:03.773 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738cda9700 1 --2- 172.21.15.195:0/3013550864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 0x7f738813a6b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f737c009cc0 tx=0x7f737c009220).ready entity=mon.1 client_cookie=b924ef48eb225134 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:03.773 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 1 -- 172.21.15.195:0/3013550864 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c00bbf0 con 0x7f73881364b0 2021-03-09T08:48:03.773 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.774 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T08:48:03.774 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 10 monclient: dump: 2021-03-09T08:48:03.774 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.774 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.775 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.775 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.775 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.775 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.776 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.776 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.776 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.777 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.777 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 10 monclient: _finish_auth 0 2021-03-09T08:48:03.777 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:03.777 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 1 -- 172.21.15.195:0/3013550864 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c00bd50 con 0x7f73881364b0 2021-03-09T08:48:03.778 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 5 monclient: authenticate success, global_id 4141 2021-03-09T08:48:03.778 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:03.778 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:03.778 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:03.779 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 1 -- 172.21.15.195:0/3013550864 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c011610 con 0x7f73881364b0 2021-03-09T08:48:03.779 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.779 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f738e80c700 4 set_mon_vals no callback set 2021-03-09T08:48:03.780 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:03.780 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.748+0000 7f7386ffd700 10 monclient: dump: 2021-03-09T08:48:03.780 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.780 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.781 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.781 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.781 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.781 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.782 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.782 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.749+0000 7f738e80c700 10 monclient: shutdown 2021-03-09T08:48:03.783 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.749+0000 7f738e80c700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:03.783 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.749+0000 7f738e80c700 1 -- 172.21.15.195:0/3013550864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 msgr2=0x7f738813a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.783 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.749+0000 7f738e80c700 1 --2- 172.21.15.195:0/3013550864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 0x7f738813a6b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f737c009cc0 tx=0x7f737c009220).stop 2021-03-09T08:48:03.783 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.749+0000 7f738e80c700 1 -- 172.21.15.195:0/3013550864 shutdown_connections 2021-03-09T08:48:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.749+0000 7f738e80c700 1 --2- 172.21.15.195:0/3013550864 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 0x7f738813a6b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.749+0000 7f738e80c700 1 -- 172.21.15.195:0/3013550864 >> 172.21.15.195:0/3013550864 conn(0x7f7388065ac0 msgr2=0x7f73880a9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.749+0000 7f738e80c700 1 -- 172.21.15.195:0/3013550864 shutdown_connections 2021-03-09T08:48:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.749+0000 7f738e80c700 1 -- 172.21.15.195:0/3013550864 wait complete. 2021-03-09T08:48:03.784 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.750+0000 7f738e80c700 10 monclient: build_initial_monmap 2021-03-09T08:48:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.750+0000 7f738e80c700 10 monclient: monmap: 2021-03-09T08:48:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:03.785 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2021-03-09T08:48:03.786 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2021-03-09T08:48:03.786 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:03.786 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2021-03-09T08:48:03.786 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:03.787 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:03.787 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:03.787 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.787 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.750+0000 7f738e80c700 1 librados: starting msgr at 2021-03-09T08:48:03.787 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.750+0000 7f738e80c700 1 librados: starting objecter 2021-03-09T08:48:03.788 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.750+0000 7f738e80c700 1 Processor -- start 2021-03-09T08:48:03.788 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.750+0000 7f738e80c700 1 -- start start 2021-03-09T08:48:03.788 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.750+0000 7f738e80c700 1 librados: setting wanted keys 2021-03-09T08:48:03.788 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.750+0000 7f738e80c700 1 librados: calling monclient init 2021-03-09T08:48:03.789 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.750+0000 7f738e80c700 10 monclient: init 2021-03-09T08:48:03.789 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.789 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:48:03.789 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7388069f90 0x7f73880a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.790 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient(hunting): picked mon.noname-a con 0x7f7388069f90 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:03.790 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f7388136050 0x7f73880a3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.790 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient(hunting): picked mon.noname-c con 0x7f7388136050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:03.790 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 0x7f7388135450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.791 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient(hunting): picked mon.noname-b con 0x7f73881364b0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:03.791 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.791 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73880a2e60 con 0x7f7388069f90 2021-03-09T08:48:03.791 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.792 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73880a2b10 con 0x7f73881364b0 2021-03-09T08:48:03.792 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:03.792 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7388068030 con 0x7f7388136050 2021-03-09T08:48:03.792 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:03.792 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient(hunting): authenticate will time out at 849.241117s 2021-03-09T08:48:03.793 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 0x7f7388135450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.793 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 0x7f7388135450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45826/0 (socket says 172.21.15.195:45826) 2021-03-09T08:48:03.793 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 -- 172.21.15.195:0/2694240516 learned_addr learned my addr 172.21.15.195:0/2694240516 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:03.793 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7387fff700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7388069f90 0x7f73880a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.794 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 10 monclient(hunting): get_auth_request con 0x7f73881364b0 auth_method 0 2021-03-09T08:48:03.794 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.794 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.794 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7387fff700 10 monclient(hunting): get_auth_request con 0x7f7388069f90 auth_method 0 2021-03-09T08:48:03.795 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7387fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.795 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7387fff700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.795 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f73877fe700 1 --2- 172.21.15.195:0/2694240516 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f7388136050 0x7f73880a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:03.795 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f73877fe700 10 monclient(hunting): get_auth_request con 0x7f7388136050 auth_method 0 2021-03-09T08:48:03.795 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f73877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:03.796 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f73877fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:03.796 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.796 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.796 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.797 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7387fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.797 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7387fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.797 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7387fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.797 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f73877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:03.797 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f73877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:03.798 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f73877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:03.798 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 10 monclient(hunting): handle_auth_done global_id 4147 payload 995 2021-03-09T08:48:03.798 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 -- 172.21.15.195:0/2694240516 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f7388136050 msgr2=0x7f73880a3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.798 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 --2- 172.21.15.195:0/2694240516 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f7388136050 0x7f73880a3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.799 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 -- 172.21.15.195:0/2694240516 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7388069f90 msgr2=0x7f73880a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.799 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 --2- 172.21.15.195:0/2694240516 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7388069f90 0x7f73880a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.799 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 10 monclient: _finish_hunting 0 2021-03-09T08:48:03.799 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 monclient: found mon.noname-b 2021-03-09T08:48:03.800 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:03.800 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.800 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 -- 172.21.15.195:0/2694240516 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73880a12f0 con 0x7f73881364b0 2021-03-09T08:48:03.800 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738cda9700 1 --2- 172.21.15.195:0/2694240516 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 0x7f7388135450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f737c006f80 tx=0x7f737c017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:03.801 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 client.?.objecter ms_handle_connect 0x7f73881364b0 2021-03-09T08:48:03.801 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:03.801 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 1 -- 172.21.15.195:0/2694240516 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c011e30 con 0x7f73881364b0 2021-03-09T08:48:03.801 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.801 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:03.802 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 monclient: dump: 2021-03-09T08:48:03.802 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.802 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.802 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.803 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.803 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.803 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.803 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.804 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.804 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.804 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.804 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 monclient: _finish_auth 0 2021-03-09T08:48:03.805 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:03.805 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 5 monclient: authenticate success, global_id 4147 2021-03-09T08:48:03.805 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 1 -- 172.21.15.195:0/2694240516 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c0109e0 con 0x7f73881364b0 2021-03-09T08:48:03.805 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient: _renew_subs 2021-03-09T08:48:03.805 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.806 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 1 -- 172.21.15.195:0/2694240516 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73880a3c40 con 0x7f73881364b0 2021-03-09T08:48:03.806 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:03.806 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 1 -- 172.21.15.195:0/2694240516 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c010da0 con 0x7f73881364b0 2021-03-09T08:48:03.806 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:03.807 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:03.807 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f7373fff700 10 monclient: dump: 2021-03-09T08:48:03.807 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:03.807 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.808 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.808 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:03.808 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:03.808 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.809 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:03.809 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:03.809 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:03.809 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.809 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:03.810 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient: _renew_subs 2021-03-09T08:48:03.810 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.810 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 1 -- 172.21.15.195:0/2694240516 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7388066430 con 0x7f73881364b0 2021-03-09T08:48:03.810 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f738e80c700 1 librados: init done 2021-03-09T08:48:03.811 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.751+0000 7f73867fc700 4 set_mon_vals no callback set 2021-03-09T08:48:03.811 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f7373fff700 1 -- 172.21.15.195:0/2694240516 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f737c02a070 con 0x7f73881364b0 2021-03-09T08:48:03.811 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f7373fff700 1 -- 172.21.15.195:0/2694240516 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7f737c01bcc0 con 0x7f73881364b0 2021-03-09T08:48:03.811 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f7373fff700 10 client.4147.objecter ms_dispatch 0x7f7388001550 osd_map(6..6 src has 1..6) v4 2021-03-09T08:48:03.812 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f7373fff700 3 client.4147.objecter handle_osd_map got epochs [6,6] > 0 2021-03-09T08:48:03.812 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f7373fff700 3 client.4147.objecter handle_osd_map decoding full epoch 6 2021-03-09T08:48:03.812 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f7373fff700 20 client.4147.objecter dump_active .. 0 homeless 2021-03-09T08:48:03.812 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f738e80c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f738e80c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f738e80c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.752+0000 7f738e80c700 1 -- 172.21.15.195:0/2694240516 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73880a8030 con 0x7f73881364b0 2021-03-09T08:48:03.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.756+0000 7f7373fff700 1 -- 172.21.15.195:0/2694240516 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f737c028060 con 0x7f73881364b0 2021-03-09T08:48:03.813 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.756+0000 7f7373fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:03.814 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.756+0000 7f7373fff700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:03.835 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.833+0000 7f738e80c700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "802be6f1-4e15-4e2d-8720-0a3009f8d02e", "id": 4}] 2021-03-09T08:48:03.835 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.833+0000 7f738e80c700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "802be6f1-4e15-4e2d-8720-0a3009f8d02e", "id": 4}] 2021-03-09T08:48:03.835 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.833+0000 7f738e80c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:03.836 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.833+0000 7f738e80c700 1 -- 172.21.15.195:0/2694240516 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "802be6f1-4e15-4e2d-8720-0a3009f8d02e", "id": 4} v 0) v1 -- 0x7f738805e040 con 0x7f73881364b0 2021-03-09T08:48:03.838 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.836+0000 7f7373fff700 1 -- 172.21.15.195:0/2694240516 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "802be6f1-4e15-4e2d-8720-0a3009f8d02e", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f73880a0aa0 con 0x7f73881364b0 2021-03-09T08:48:03.838 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.836+0000 7f7373fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "802be6f1-4e15-4e2d-8720-0a3009f8d02e", "id": 4}] 2021-03-09T08:48:03.839 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.836+0000 7f7373fff700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:03.839 INFO:teuthology.orchestra.run.smithi195.stdout:4 2021-03-09T08:48:03.841 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 10 librados: watch_flush enter 2021-03-09T08:48:03.841 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 10 librados: watch_flush exit 2021-03-09T08:48:03.842 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 20 client.4147.objecter shutdown clearing up homeless session... 2021-03-09T08:48:03.842 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 10 client.4147.objecter successfully canceled tick 2021-03-09T08:48:03.842 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 10 monclient: shutdown 2021-03-09T08:48:03.842 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:03.843 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 1 -- 172.21.15.195:0/2694240516 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 msgr2=0x7f7388135450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:03.843 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 1 --2- 172.21.15.195:0/2694240516 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 0x7f7388135450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f737c006f80 tx=0x7f737c017000).stop 2021-03-09T08:48:03.843 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 1 -- 172.21.15.195:0/2694240516 shutdown_connections 2021-03-09T08:48:03.843 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 1 --2- 172.21.15.195:0/2694240516 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f73881364b0 0x7f7388135450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.844 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 1 --2- 172.21.15.195:0/2694240516 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f7388069f90 0x7f73880a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.844 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 1 --2- 172.21.15.195:0/2694240516 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f7388136050 0x7f73880a3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:03.844 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.837+0000 7f738e80c700 1 -- 172.21.15.195:0/2694240516 >> 172.21.15.195:0/2694240516 conn(0x7f7388065ac0 msgr2=0x7f738809f590 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:03.844 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.838+0000 7f738e80c700 1 -- 172.21.15.195:0/2694240516 shutdown_connections 2021-03-09T08:48:03.845 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.838+0000 7f738e80c700 1 -- 172.21.15.195:0/2694240516 wait complete. 2021-03-09T08:48:03.845 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.838+0000 7f738e80c700 1 librados: shutdown 2021-03-09T08:48:03.851 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 17c2bea4-c1fc-49d2-a191-957e9575fd9e 5 2021-03-09T08:48:03.982 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.980+0000 7f6d1f3d2700 10 monclient: get_monmap_and_config 2021-03-09T08:48:03.983 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.980+0000 7f6d1f3d2700 10 monclient: build_initial_monmap 2021-03-09T08:48:03.983 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.980+0000 7f6d1f3d2700 10 monclient: monmap: 2021-03-09T08:48:03.984 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:03.984 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:03.984 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:48:03.981278+0000 2021-03-09T08:48:03.984 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:48:03.981278+0000 2021-03-09T08:48:03.984 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:03.985 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:03.985 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:03.985 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:03.985 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:03.986 INFO:teuthology.orchestra.run.smithi195.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:03.986 INFO:teuthology.orchestra.run.smithi195.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:03.986 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:03.986 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 1 Processor -- start 2021-03-09T08:48:03.987 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 1 -- start start 2021-03-09T08:48:03.987 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient: init 2021-03-09T08:48:03.999 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:03.999 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2021-03-09T08:48:03.999 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6d180a5fd0 0x7f6d180a4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:03.999 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient(hunting): picked mon.noname-c con 0x7f6d180a5fd0 addr v2:172.21.15.195:3300/0 2021-03-09T08:48:04.000 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d180a3950 0x7f6d1812fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.000 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient(hunting): picked mon.noname-b con 0x7f6d180a3950 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:04.000 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f6d181302d0 0x7f6d18133090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.000 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient(hunting): picked mon.noname-a con 0x7f6d181302d0 addr v2:172.21.15.99:3300/0 2021-03-09T08:48:04.001 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.001 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d180a4990 con 0x7f6d181302d0 2021-03-09T08:48:04.001 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.001 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1809e450 con 0x7f6d180a5fd0 2021-03-09T08:48:04.002 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.002 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d180a63b0 con 0x7f6d180a3950 2021-03-09T08:48:04.002 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:04.002 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.981+0000 7f6d1f3d2700 10 monclient(hunting): authenticate will time out at 849.471868s 2021-03-09T08:48:04.003 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6d180a5fd0 0x7f6d180a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.003 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6d180a5fd0 0x7f6d180a4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45828/0 (socket says 172.21.15.195:45828) 2021-03-09T08:48:04.003 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 -- 172.21.15.195:0/1239891618 learned_addr learned my addr 172.21.15.195:0/1239891618 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:04.003 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 10 monclient(hunting): get_auth_request con 0x7f6d180a5fd0 auth_method 0 2021-03-09T08:48:04.003 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.004 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.004 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.004 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.004 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.005 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1ebd1700 1 --2- 172.21.15.195:0/1239891618 >> v2:172.21.15.99:3300/0 conn(0x7f6d181302d0 0x7f6d18133090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.005 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1ebd1700 10 monclient(hunting): get_auth_request con 0x7f6d181302d0 auth_method 0 2021-03-09T08:48:04.005 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1ebd1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.005 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1ebd1700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.006 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1dbcf700 1 --2- 172.21.15.195:0/1239891618 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d180a3950 0x7f6d1812fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.006 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1dbcf700 10 monclient(hunting): get_auth_request con 0x7f6d180a3950 auth_method 0 2021-03-09T08:48:04.006 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1dbcf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.006 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1dbcf700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.007 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1ebd1700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.007 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1ebd1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.007 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1ebd1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.007 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 10 monclient(hunting): handle_auth_done global_id 4150 payload 450 2021-03-09T08:48:04.008 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 -- 172.21.15.195:0/1239891618 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d180a3950 msgr2=0x7f6d1812fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:48:04.008 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 --2- 172.21.15.195:0/1239891618 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d180a3950 0x7f6d1812fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.008 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 -- 172.21.15.195:0/1239891618 >> v2:172.21.15.99:3300/0 conn(0x7f6d181302d0 msgr2=0x7f6d18133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:48:04.008 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 --2- 172.21.15.195:0/1239891618 >> v2:172.21.15.99:3300/0 conn(0x7f6d181302d0 0x7f6d18133090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.009 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 10 monclient: _finish_hunting 0 2021-03-09T08:48:04.009 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 monclient: found mon.noname-c 2021-03-09T08:48:04.009 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:04.009 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.010 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 -- 172.21.15.195:0/1239891618 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d18130780 con 0x7f6d180a5fd0 2021-03-09T08:48:04.010 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1dbcf700 1 --2- 172.21.15.195:0/1239891618 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d180a3950 0x7f6d1812fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2021-03-09T08:48:04.010 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1e3d0700 1 --2- 172.21.15.195:0/1239891618 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a5fd0 0x7f6d180a4240 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f6d180a6550 tx=0x7f6d08006ee0).ready entity=mon.1 client_cookie=87a10fe59fb35f81 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:04.010 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 1 -- 172.21.15.195:0/1239891618 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08009840 con 0x7f6d180a5fd0 2021-03-09T08:48:04.011 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.011 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T08:48:04.011 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 10 monclient: dump: 2021-03-09T08:48:04.011 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.011 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.012 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.012 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.012 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.012 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.013 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.013 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.013 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.013 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.013 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 10 monclient: _finish_auth 0 2021-03-09T08:48:04.014 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:04.014 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1f3d2700 5 monclient: authenticate success, global_id 4150 2021-03-09T08:48:04.014 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1f3d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:04.014 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 1 -- 172.21.15.195:0/1239891618 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d080099a0 con 0x7f6d180a5fd0 2021-03-09T08:48:04.015 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:04.015 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1f3d2700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:04.015 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1f3d2700 4 set_mon_vals no callback set 2021-03-09T08:48:04.015 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 1 -- 172.21.15.195:0/1239891618 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08009d20 con 0x7f6d180a5fd0 2021-03-09T08:48:04.016 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.016 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:04.016 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.982+0000 7f6d1d3ce700 10 monclient: dump: 2021-03-09T08:48:04.016 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.017 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.017 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.017 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.017 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.017 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.018 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.018 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.018 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.018 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.019 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 10 monclient: shutdown 2021-03-09T08:48:04.019 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:04.019 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/1239891618 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a5fd0 msgr2=0x7f6d180a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.019 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 1 --2- 172.21.15.195:0/1239891618 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a5fd0 0x7f6d180a4240 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f6d180a6550 tx=0x7f6d08006ee0).stop 2021-03-09T08:48:04.020 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/1239891618 shutdown_connections 2021-03-09T08:48:04.020 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 1 --2- 172.21.15.195:0/1239891618 >> v2:172.21.15.99:3300/0 conn(0x7f6d181302d0 0x7f6d18133090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.020 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 1 --2- 172.21.15.195:0/1239891618 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a5fd0 0x7f6d180a4240 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.020 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 1 --2- 172.21.15.195:0/1239891618 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d180a3950 0x7f6d1812fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.020 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/1239891618 >> 172.21.15.195:0/1239891618 conn(0x7f6d180642c0 msgr2=0x7f6d18069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:04.021 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/1239891618 shutdown_connections 2021-03-09T08:48:04.021 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/1239891618 wait complete. 2021-03-09T08:48:04.021 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 10 monclient: build_initial_monmap 2021-03-09T08:48:04.021 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.983+0000 7f6d1f3d2700 10 monclient: monmap: 2021-03-09T08:48:04.022 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:04.022 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:04.022 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2021-03-09T08:48:04.022 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2021-03-09T08:48:04.023 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:04.023 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2021-03-09T08:48:04.023 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:04.023 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:04.023 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:04.024 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.024 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 librados: starting msgr at 2021-03-09T08:48:04.024 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 librados: starting objecter 2021-03-09T08:48:04.024 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 Processor -- start 2021-03-09T08:48:04.025 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 -- start start 2021-03-09T08:48:04.025 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 librados: setting wanted keys 2021-03-09T08:48:04.025 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 librados: calling monclient init 2021-03-09T08:48:04.025 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient: init 2021-03-09T08:48:04.026 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:04.026 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:48:04.026 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a3950 0x7f6d18135840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.026 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient(hunting): picked mon.noname-b con 0x7f6d180a3950 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:04.026 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6d180a5fd0 0x7f6d181499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.027 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient(hunting): picked mon.noname-a con 0x7f6d180a5fd0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:04.027 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d181302d0 0x7f6d1813ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.027 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient(hunting): picked mon.noname-c con 0x7f6d181302d0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:04.027 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.028 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18135dc0 con 0x7f6d180a5fd0 2021-03-09T08:48:04.028 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.028 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18149540 con 0x7f6d180a3950 2021-03-09T08:48:04.028 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.029 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d18149ed0 con 0x7f6d181302d0 2021-03-09T08:48:04.029 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:04.029 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1f3d2700 10 monclient(hunting): authenticate will time out at 849.474776s 2021-03-09T08:48:04.029 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1e3d0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a3950 0x7f6d18135840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.029 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1e3d0700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a3950 0x7f6d18135840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45834/0 (socket says 172.21.15.195:45834) 2021-03-09T08:48:04.030 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1e3d0700 1 -- 172.21.15.195:0/2039589428 learned_addr learned my addr 172.21.15.195:0/2039589428 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:04.030 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1e3d0700 10 monclient(hunting): get_auth_request con 0x7f6d180a3950 auth_method 0 2021-03-09T08:48:04.030 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1e3d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.030 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1e3d0700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.031 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1ebd1700 1 --2- 172.21.15.195:0/2039589428 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d181302d0 0x7f6d1813ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.031 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1dbcf700 1 --2- 172.21.15.195:0/2039589428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6d180a5fd0 0x7f6d181499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.031 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1e3d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.031 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.984+0000 7f6d1e3d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.032 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.032 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1ebd1700 10 monclient(hunting): get_auth_request con 0x7f6d181302d0 auth_method 0 2021-03-09T08:48:04.032 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1ebd1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.032 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1ebd1700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.032 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1dbcf700 10 monclient(hunting): get_auth_request con 0x7f6d180a5fd0 auth_method 0 2021-03-09T08:48:04.033 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1dbcf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.033 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1dbcf700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.033 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1ebd1700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.033 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1ebd1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.034 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1ebd1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.034 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1dbcf700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.034 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1dbcf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.034 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1dbcf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.035 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 10 monclient(hunting): handle_auth_done global_id 4153 payload 995 2021-03-09T08:48:04.035 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 1 -- 172.21.15.195:0/2039589428 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d181302d0 msgr2=0x7f6d1813ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.035 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 1 --2- 172.21.15.195:0/2039589428 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d181302d0 0x7f6d1813ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.035 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 1 -- 172.21.15.195:0/2039589428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6d180a5fd0 msgr2=0x7f6d181499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.036 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 1 --2- 172.21.15.195:0/2039589428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6d180a5fd0 0x7f6d181499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.036 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 10 monclient: _finish_hunting 0 2021-03-09T08:48:04.036 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 1 monclient: found mon.noname-b 2021-03-09T08:48:04.036 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:04.036 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.037 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 1 -- 172.21.15.195:0/2039589428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d1812f7e0 con 0x7f6d180a3950 2021-03-09T08:48:04.037 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1e3d0700 1 --2- 172.21.15.195:0/2039589428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a3950 0x7f6d18135840 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f6d080093a0 tx=0x7f6d08017000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:04.037 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 client.?.objecter ms_handle_connect 0x7f6d180a3950 2021-03-09T08:48:04.037 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:04.038 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 1 -- 172.21.15.195:0/2039589428 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08010d50 con 0x7f6d180a3950 2021-03-09T08:48:04.038 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.038 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:04.038 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 monclient: dump: 2021-03-09T08:48:04.039 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.039 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.039 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.039 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.040 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.040 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.040 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.040 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.041 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.041 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.041 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 monclient: _finish_auth 0 2021-03-09T08:48:04.041 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:04.042 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 1 -- 172.21.15.195:0/2039589428 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d08024420 con 0x7f6d180a3950 2021-03-09T08:48:04.042 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1f3d2700 5 monclient: authenticate success, global_id 4153 2021-03-09T08:48:04.042 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1f3d2700 10 monclient: _renew_subs 2021-03-09T08:48:04.042 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1f3d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.043 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/2039589428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1813bef0 con 0x7f6d180a3950 2021-03-09T08:48:04.043 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:04.043 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 1 -- 172.21.15.195:0/2039589428 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d080245d0 con 0x7f6d180a3950 2021-03-09T08:48:04.043 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.044 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:04.044 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 monclient: dump: 2021-03-09T08:48:04.044 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.045 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.045 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.045 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.046 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.046 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.046 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.047 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.047 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.047 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.047 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1cbcd700 4 set_mon_vals no callback set 2021-03-09T08:48:04.048 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1f3d2700 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:04.048 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1f3d2700 10 monclient: _renew_subs 2021-03-09T08:48:04.048 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1f3d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.049 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/2039589428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d181492b0 con 0x7f6d180a3950 2021-03-09T08:48:04.049 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d1f3d2700 1 librados: init done 2021-03-09T08:48:04.049 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 1 -- 172.21.15.195:0/2039589428 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f6d0802b070 con 0x7f6d180a3950 2021-03-09T08:48:04.050 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 1 -- 172.21.15.195:0/2039589428 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f6d0801bbf0 con 0x7f6d180a3950 2021-03-09T08:48:04.050 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 10 client.4153.objecter ms_dispatch 0x7f6d18001550 osd_map(7..7 src has 1..7) v4 2021-03-09T08:48:04.050 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 3 client.4153.objecter handle_osd_map got epochs [7,7] > 0 2021-03-09T08:48:04.051 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 3 client.4153.objecter handle_osd_map decoding full epoch 7 2021-03-09T08:48:04.051 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.985+0000 7f6d067fc700 20 client.4153.objecter dump_active .. 0 homeless 2021-03-09T08:48:04.051 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.986+0000 7f6d1f3d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:04.052 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.986+0000 7f6d1f3d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:04.052 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.986+0000 7f6d1f3d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.052 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.986+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/2039589428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d180a7700 con 0x7f6d180a3950 2021-03-09T08:48:04.052 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.988+0000 7f6d067fc700 1 -- 172.21.15.195:0/2039589428 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f6d08043980 con 0x7f6d180a3950 2021-03-09T08:48:04.053 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.988+0000 7f6d067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:04.053 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:03.988+0000 7f6d067fc700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:04.069 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.066+0000 7f6d1f3d2700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "17c2bea4-c1fc-49d2-a191-957e9575fd9e", "id": 5}] 2021-03-09T08:48:04.071 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.066+0000 7f6d1f3d2700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "17c2bea4-c1fc-49d2-a191-957e9575fd9e", "id": 5}] 2021-03-09T08:48:04.071 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.066+0000 7f6d1f3d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.071 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.066+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/2039589428 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "17c2bea4-c1fc-49d2-a191-957e9575fd9e", "id": 5} v 0) v1 -- 0x7f6d180589f0 con 0x7f6d180a3950 2021-03-09T08:48:04.072 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.069+0000 7f6d067fc700 1 -- 172.21.15.195:0/2039589428 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "17c2bea4-c1fc-49d2-a191-957e9575fd9e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6d08032030 con 0x7f6d180a3950 2021-03-09T08:48:04.072 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.069+0000 7f6d067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "17c2bea4-c1fc-49d2-a191-957e9575fd9e", "id": 5}] 2021-03-09T08:48:04.072 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.069+0000 7f6d067fc700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:04.073 INFO:teuthology.orchestra.run.smithi195.stdout:5 2021-03-09T08:48:04.074 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 10 librados: watch_flush enter 2021-03-09T08:48:04.075 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 10 librados: watch_flush exit 2021-03-09T08:48:04.075 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 20 client.4153.objecter shutdown clearing up homeless session... 2021-03-09T08:48:04.075 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 10 client.4153.objecter successfully canceled tick 2021-03-09T08:48:04.075 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 10 monclient: shutdown 2021-03-09T08:48:04.076 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:04.076 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/2039589428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a3950 msgr2=0x7f6d18135840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.076 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 --2- 172.21.15.195:0/2039589428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a3950 0x7f6d18135840 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f6d080093a0 tx=0x7f6d08017000).stop 2021-03-09T08:48:04.076 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/2039589428 shutdown_connections 2021-03-09T08:48:04.076 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 --2- 172.21.15.195:0/2039589428 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6d181302d0 0x7f6d1813ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.077 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 --2- 172.21.15.195:0/2039589428 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6d180a3950 0x7f6d18135840 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.077 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 --2- 172.21.15.195:0/2039589428 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f6d180a5fd0 0x7f6d181499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.077 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/2039589428 >> 172.21.15.195:0/2039589428 conn(0x7f6d180642c0 msgr2=0x7f6d180a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:04.077 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/2039589428 shutdown_connections 2021-03-09T08:48:04.077 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 -- 172.21.15.195:0/2039589428 wait complete. 2021-03-09T08:48:04.078 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.070+0000 7f6d1f3d2700 1 librados: shutdown 2021-03-09T08:48:04.083 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 27bb6c24-657c-4887-b344-aa7c2dd5676f 6 2021-03-09T08:48:04.216 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.214+0000 7f8d3359e700 10 monclient: get_monmap_and_config 2021-03-09T08:48:04.217 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.214+0000 7f8d3359e700 10 monclient: build_initial_monmap 2021-03-09T08:48:04.217 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.214+0000 7f8d3359e700 10 monclient: monmap: 2021-03-09T08:48:04.217 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:04.218 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.218 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:48:04.215239+0000 2021-03-09T08:48:04.218 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:48:04.215239+0000 2021-03-09T08:48:04.218 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:04.218 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.219 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:04.219 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:04.219 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:04.219 INFO:teuthology.orchestra.run.smithi195.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:04.220 INFO:teuthology.orchestra.run.smithi195.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:04.220 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.220 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 1 Processor -- start 2021-03-09T08:48:04.220 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 1 -- start start 2021-03-09T08:48:04.220 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 10 monclient: init 2021-03-09T08:48:04.231 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:04.231 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2021-03-09T08:48:04.231 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d3409f6b0 addr v1:172.21.15.99:6789/0 2021-03-09T08:48:04.232 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8d3409fa90 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:04.232 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8d3413f1b0 0x7f8d341415a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.232 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 10 monclient(hunting): picked mon.noname-c con 0x7f8d3413f1b0 addr v2:172.21.15.195:3300/0 2021-03-09T08:48:04.232 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d340a2740 con 0x7f8d3409f6b0 2021-03-09T08:48:04.233 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d340a0810 con 0x7f8d3409fa90 2021-03-09T08:48:04.233 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.233 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d340a09a0 con 0x7f8d3413f1b0 2021-03-09T08:48:04.233 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:04.234 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d3359e700 10 monclient(hunting): authenticate will time out at 849.705851s 2021-03-09T08:48:04.234 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d32d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8d3413f1b0 0x7f8d341415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.234 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.215+0000 7f8d31d9b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f8d3409fa90 0x7f8d3413a390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48054/0 (socket says 172.21.15.195:48054) 2021-03-09T08:48:04.234 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f8d3413f1b0 0x7f8d341415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45844/0 (socket says 172.21.15.195:45844) 2021-03-09T08:48:04.234 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d31d9b700 1 -- 172.21.15.195:0/2676594093 learned_addr learned my addr 172.21.15.195:0/2676594093 (peer_addr_for_me v1:172.21.15.195:0/0) 2021-03-09T08:48:04.235 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 10 monclient(hunting): get_auth_request con 0x7f8d3413f1b0 auth_method 0 2021-03-09T08:48:04.235 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.235 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.235 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.235 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.236 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.236 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 10 monclient(hunting): handle_auth_done global_id 4156 payload 450 2021-03-09T08:48:04.236 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 1 -- 172.21.15.195:0/2676594093 >> v1:172.21.15.195:6789/0 conn(0x7f8d3409fa90 legacy=0x7f8d3413a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.236 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 1 -- 172.21.15.195:0/2676594093 >> v1:172.21.15.99:6789/0 conn(0x7f8d3409f6b0 legacy=0x7f8d34139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.237 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 10 monclient: _finish_hunting 0 2021-03-09T08:48:04.237 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 1 monclient: found mon.noname-c 2021-03-09T08:48:04.237 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:04.237 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.237 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 1 -- 172.21.15.195:0/2676594093 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d340a2370 con 0x7f8d3413f1b0 2021-03-09T08:48:04.238 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d32d9d700 1 --2- 172.21.15.195:0/2676594093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3413f1b0 0x7f8d341415a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f8d24009cc0 tx=0x7f8d24009220).ready entity=mon.1 client_cookie=4f5f40aa2fa10727 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:04.238 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 1 -- 172.21.15.195:0/2676594093 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2400bbf0 con 0x7f8d3413f1b0 2021-03-09T08:48:04.238 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.238 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T08:48:04.238 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 10 monclient: dump: 2021-03-09T08:48:04.239 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.239 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.239 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.239 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.240 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.240 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.240 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.240 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.240 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.241 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.241 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 10 monclient: _finish_auth 0 2021-03-09T08:48:04.241 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:04.241 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 1 -- 172.21.15.195:0/2676594093 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2400bd50 con 0x7f8d3413f1b0 2021-03-09T08:48:04.241 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3359e700 5 monclient: authenticate success, global_id 4156 2021-03-09T08:48:04.242 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3359e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:04.242 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:04.242 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3359e700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:04.242 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 1 -- 172.21.15.195:0/2676594093 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d24011610 con 0x7f8d3413f1b0 2021-03-09T08:48:04.242 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3359e700 4 set_mon_vals no callback set 2021-03-09T08:48:04.243 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.243 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:04.243 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3159a700 10 monclient: dump: 2021-03-09T08:48:04.243 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.244 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.244 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.244 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.244 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.244 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.245 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.245 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.245 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.245 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.245 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3359e700 10 monclient: shutdown 2021-03-09T08:48:04.246 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3359e700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:04.246 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.216+0000 7f8d3359e700 1 -- 172.21.15.195:0/2676594093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3413f1b0 msgr2=0x7f8d341415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.246 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.217+0000 7f8d3359e700 1 --2- 172.21.15.195:0/2676594093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3413f1b0 0x7f8d341415a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f8d24009cc0 tx=0x7f8d24009220).stop 2021-03-09T08:48:04.247 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.217+0000 7f8d3359e700 1 -- 172.21.15.195:0/2676594093 shutdown_connections 2021-03-09T08:48:04.247 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.217+0000 7f8d3359e700 1 --2- 172.21.15.195:0/2676594093 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3413f1b0 0x7f8d341415a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.247 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.217+0000 7f8d3359e700 1 -- 172.21.15.195:0/2676594093 >> 172.21.15.195:0/2676594093 conn(0x7f8d34063240 msgr2=0x7f8d34068510 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:04.247 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.217+0000 7f8d3359e700 1 -- 172.21.15.195:0/2676594093 shutdown_connections 2021-03-09T08:48:04.248 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.217+0000 7f8d3359e700 1 -- 172.21.15.195:0/2676594093 wait complete. 2021-03-09T08:48:04.248 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.217+0000 7f8d3359e700 10 monclient: build_initial_monmap 2021-03-09T08:48:04.248 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.217+0000 7f8d3359e700 10 monclient: monmap: 2021-03-09T08:48:04.248 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:04.248 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:04.249 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2021-03-09T08:48:04.249 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2021-03-09T08:48:04.249 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:04.249 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2021-03-09T08:48:04.250 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:04.250 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:04.250 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:04.251 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.251 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.218+0000 7f8d3359e700 1 librados: starting msgr at 2021-03-09T08:48:04.251 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.218+0000 7f8d3359e700 1 librados: starting objecter 2021-03-09T08:48:04.251 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.218+0000 7f8d3359e700 1 Processor -- start 2021-03-09T08:48:04.252 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.218+0000 7f8d3359e700 1 -- start start 2021-03-09T08:48:04.252 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.218+0000 7f8d3359e700 1 librados: setting wanted keys 2021-03-09T08:48:04.252 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.218+0000 7f8d3359e700 1 librados: calling monclient init 2021-03-09T08:48:04.253 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.218+0000 7f8d3359e700 10 monclient: init 2021-03-09T08:48:04.253 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:04.253 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:48:04.253 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d3409f6b0 0x7f8d34139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.254 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient(hunting): picked mon.noname-a con 0x7f8d3409f6b0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:04.254 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3409fa90 0x7f8d3413d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.254 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient(hunting): picked mon.noname-b con 0x7f8d3409fa90 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:04.255 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f8d3413f1b0 0x7f8d3412e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.255 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient(hunting): picked mon.noname-c con 0x7f8d3413f1b0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:04.255 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.255 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3413da30 con 0x7f8d3409f6b0 2021-03-09T08:48:04.256 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.256 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3413ded0 con 0x7f8d3409fa90 2021-03-09T08:48:04.256 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.256 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3413d560 con 0x7f8d3413f1b0 2021-03-09T08:48:04.256 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:04.257 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3359e700 10 monclient(hunting): authenticate will time out at 849.709502s 2021-03-09T08:48:04.257 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3409fa90 0x7f8d3413d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.257 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3409fa90 0x7f8d3413d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45848/0 (socket says 172.21.15.195:45848) 2021-03-09T08:48:04.257 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 -- 172.21.15.195:0/1058614523 learned_addr learned my addr 172.21.15.195:0/1058614523 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:04.258 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 10 monclient(hunting): get_auth_request con 0x7f8d3409fa90 auth_method 0 2021-03-09T08:48:04.258 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.258 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.258 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.259 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.259 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.259 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d32d9d700 1 --2- 172.21.15.195:0/1058614523 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f8d3413f1b0 0x7f8d3412e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.259 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3259c700 1 --2- 172.21.15.195:0/1058614523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d3409f6b0 0x7f8d34139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.259 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d32d9d700 10 monclient(hunting): get_auth_request con 0x7f8d3413f1b0 auth_method 0 2021-03-09T08:48:04.260 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d32d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.260 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d32d9d700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.260 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3259c700 10 monclient(hunting): get_auth_request con 0x7f8d3409f6b0 auth_method 0 2021-03-09T08:48:04.260 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.261 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3259c700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.261 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.261 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.261 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d3259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.261 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d32d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.262 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d32d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.262 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d32d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.262 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 10 monclient(hunting): handle_auth_done global_id 4162 payload 995 2021-03-09T08:48:04.262 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 -- 172.21.15.195:0/1058614523 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f8d3413f1b0 msgr2=0x7f8d3412e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.262 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 --2- 172.21.15.195:0/1058614523 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f8d3413f1b0 0x7f8d3412e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.263 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 -- 172.21.15.195:0/1058614523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d3409f6b0 msgr2=0x7f8d34139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.263 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 --2- 172.21.15.195:0/1058614523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d3409f6b0 0x7f8d34139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.263 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 10 monclient: _finish_hunting 0 2021-03-09T08:48:04.263 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 monclient: found mon.noname-b 2021-03-09T08:48:04.263 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:04.264 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.264 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 -- 172.21.15.195:0/1058614523 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3413d720 con 0x7f8d3409fa90 2021-03-09T08:48:04.264 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.219+0000 7f8d31d9b700 1 --2- 172.21.15.195:0/1058614523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3409fa90 0x7f8d3413d070 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8d1c001eb0 tx=0x7f8d1c007c70).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:04.264 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 client.?.objecter ms_handle_connect 0x7f8d3409fa90 2021-03-09T08:48:04.265 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:04.265 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 1 -- 172.21.15.195:0/1058614523 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c0072e0 con 0x7f8d3409fa90 2021-03-09T08:48:04.265 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.265 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:04.265 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 monclient: dump: 2021-03-09T08:48:04.266 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.266 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.266 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.266 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.266 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.267 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.267 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.267 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.267 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.268 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.268 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 monclient: _finish_auth 0 2021-03-09T08:48:04.268 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:04.268 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 1 -- 172.21.15.195:0/1058614523 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c007440 con 0x7f8d3409fa90 2021-03-09T08:48:04.268 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 5 monclient: authenticate success, global_id 4162 2021-03-09T08:48:04.269 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 10 monclient: _renew_subs 2021-03-09T08:48:04.269 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.269 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 1 -- 172.21.15.195:0/1058614523 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3412e6b0 con 0x7f8d3409fa90 2021-03-09T08:48:04.269 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:04.269 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 1 -- 172.21.15.195:0/1058614523 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c005830 con 0x7f8d3409fa90 2021-03-09T08:48:04.270 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.270 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:04.270 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 monclient: dump: 2021-03-09T08:48:04.270 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.270 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.271 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.271 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.271 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.271 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.272 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.272 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.272 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.272 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.272 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:04.273 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 10 monclient: _renew_subs 2021-03-09T08:48:04.273 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.273 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 1 -- 172.21.15.195:0/1058614523 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3412ebc0 con 0x7f8d3409fa90 2021-03-09T08:48:04.273 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 1 librados: init done 2021-03-09T08:48:04.273 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d30d99700 4 set_mon_vals no callback set 2021-03-09T08:48:04.274 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 1 -- 172.21.15.195:0/1058614523 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f8d1c01a060 con 0x7f8d3409fa90 2021-03-09T08:48:04.274 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 1 -- 172.21.15.195:0/1058614523 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f8d1c013450 con 0x7f8d3409fa90 2021-03-09T08:48:04.274 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 10 client.4162.objecter ms_dispatch 0x7f8d34001550 osd_map(8..8 src has 1..8) v4 2021-03-09T08:48:04.274 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 3 client.4162.objecter handle_osd_map got epochs [8,8] > 0 2021-03-09T08:48:04.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 3 client.4162.objecter handle_osd_map decoding full epoch 8 2021-03-09T08:48:04.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d227fc700 20 client.4162.objecter dump_active .. 0 homeless 2021-03-09T08:48:04.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:04.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:04.275 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.276 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.220+0000 7f8d3359e700 1 -- 172.21.15.195:0/1058614523 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d340589f0 con 0x7f8d3409fa90 2021-03-09T08:48:04.276 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.223+0000 7f8d227fc700 1 -- 172.21.15.195:0/1058614523 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f8d1c034980 con 0x7f8d3409fa90 2021-03-09T08:48:04.276 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.223+0000 7f8d227fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:04.276 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.223+0000 7f8d227fc700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:04.301 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.299+0000 7f8d3359e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "27bb6c24-657c-4887-b344-aa7c2dd5676f", "id": 6}] 2021-03-09T08:48:04.302 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.299+0000 7f8d3359e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "27bb6c24-657c-4887-b344-aa7c2dd5676f", "id": 6}] 2021-03-09T08:48:04.302 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.299+0000 7f8d3359e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.302 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.299+0000 7f8d3359e700 1 -- 172.21.15.195:0/1058614523 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "27bb6c24-657c-4887-b344-aa7c2dd5676f", "id": 6} v 0) v1 -- 0x7f8d34141a90 con 0x7f8d3409fa90 2021-03-09T08:48:04.306 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.302+0000 7f8d227fc700 1 -- 172.21.15.195:0/1058614523 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "27bb6c24-657c-4887-b344-aa7c2dd5676f", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8d1c023070 con 0x7f8d3409fa90 2021-03-09T08:48:04.307 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.302+0000 7f8d227fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "27bb6c24-657c-4887-b344-aa7c2dd5676f", "id": 6}] 2021-03-09T08:48:04.307 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.302+0000 7f8d227fc700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:04.309 INFO:teuthology.orchestra.run.smithi195.stdout:6 2021-03-09T08:48:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 10 librados: watch_flush enter 2021-03-09T08:48:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 10 librados: watch_flush exit 2021-03-09T08:48:04.311 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 20 client.4162.objecter shutdown clearing up homeless session... 2021-03-09T08:48:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 10 client.4162.objecter successfully canceled tick 2021-03-09T08:48:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 10 monclient: shutdown 2021-03-09T08:48:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:04.312 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 -- 172.21.15.195:0/1058614523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3409fa90 msgr2=0x7f8d3413d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.313 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 --2- 172.21.15.195:0/1058614523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3409fa90 0x7f8d3413d070 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f8d1c001eb0 tx=0x7f8d1c007c70).stop 2021-03-09T08:48:04.314 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 -- 172.21.15.195:0/1058614523 shutdown_connections 2021-03-09T08:48:04.314 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 --2- 172.21.15.195:0/1058614523 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f8d3413f1b0 0x7f8d3412e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.314 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 --2- 172.21.15.195:0/1058614523 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f8d3409f6b0 0x7f8d34139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.315 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 --2- 172.21.15.195:0/1058614523 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f8d3409fa90 0x7f8d3413d070 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.315 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 -- 172.21.15.195:0/1058614523 >> 172.21.15.195:0/1058614523 conn(0x7f8d34063240 msgr2=0x7f8d340ad650 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:04.315 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 -- 172.21.15.195:0/1058614523 shutdown_connections 2021-03-09T08:48:04.315 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 -- 172.21.15.195:0/1058614523 wait complete. 2021-03-09T08:48:04.315 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.303+0000 7f8d3359e700 1 librados: shutdown 2021-03-09T08:48:04.317 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new b11e00a8-aaeb-49b0-89a4-9720edf16c13 7 2021-03-09T08:48:04.469 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.452+0000 7f2acb43e700 10 monclient: get_monmap_and_config 2021-03-09T08:48:04.470 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.452+0000 7f2acb43e700 10 monclient: build_initial_monmap 2021-03-09T08:48:04.470 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.452+0000 7f2acb43e700 10 monclient: monmap: 2021-03-09T08:48:04.470 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:04.471 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.471 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:48:04.452942+0000 2021-03-09T08:48:04.471 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:48:04.452942+0000 2021-03-09T08:48:04.471 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:04.472 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.472 INFO:teuthology.orchestra.run.smithi195.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:04.472 INFO:teuthology.orchestra.run.smithi195.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:04.472 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:04.472 INFO:teuthology.orchestra.run.smithi195.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:04.473 INFO:teuthology.orchestra.run.smithi195.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:04.473 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.473 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.452+0000 7f2acb43e700 1 Processor -- start 2021-03-09T08:48:04.473 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.452+0000 7f2acb43e700 1 -- start start 2021-03-09T08:48:04.473 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.452+0000 7f2acb43e700 10 monclient: init 2021-03-09T08:48:04.474 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:04.474 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2021-03-09T08:48:04.474 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2ac40a5fd0 0x7f2ac40a4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.474 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 10 monclient(hunting): picked mon.noname-c con 0x7f2ac40a5fd0 addr v2:172.21.15.195:3300/0 2021-03-09T08:48:04.475 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ac40a3950 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:04.475 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ac41304f0 addr v1:172.21.15.99:6789/0 2021-03-09T08:48:04.475 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac409e450 con 0x7f2ac41304f0 2021-03-09T08:48:04.475 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ac40a63b0 con 0x7f2ac40a3950 2021-03-09T08:48:04.476 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.476 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ac40a4990 con 0x7f2ac40a5fd0 2021-03-09T08:48:04.476 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:04.476 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2acb43e700 10 monclient(hunting): authenticate will time out at 849.943567s 2021-03-09T08:48:04.476 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2aca43c700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f2ac40a5fd0 0x7f2ac40a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.477 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2ac9c3b700 1 --1- >> v1:172.21.15.195:6789/0 conn(0x7f2ac40a3950 0x7f2ac412fde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.195:6789/0 says I am v1:172.21.15.195:48066/0 (socket says 172.21.15.195:48066) 2021-03-09T08:48:04.477 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2aca43c700 10 monclient(hunting): get_auth_request con 0x7f2ac40a5fd0 auth_method 0 2021-03-09T08:48:04.477 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2ac9c3b700 1 -- 172.21.15.195:0/1232625548 learned_addr learned my addr 172.21.15.195:0/1232625548 (peer_addr_for_me v1:172.21.15.195:0/0) 2021-03-09T08:48:04.477 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2aca43c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.478 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2aca43c700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.478 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2aca43c700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.478 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2aca43c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.478 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.453+0000 7f2aca43c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.478 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2aca43c700 10 monclient(hunting): handle_auth_done global_id 4165 payload 450 2021-03-09T08:48:04.479 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2aca43c700 1 -- 172.21.15.195:0/1232625548 >> v1:172.21.15.195:6789/0 conn(0x7f2ac40a3950 legacy=0x7f2ac412fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.479 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 1 -- 172.21.15.195:0/1232625548 <== mon.0 v1:172.21.15.99:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900585558 0 0) 0x7f2ac409e450 con 0x7f2ac41304f0 2021-03-09T08:48:04.479 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2aca43c700 1 -- 172.21.15.195:0/1232625548 >> v1:172.21.15.99:6789/0 conn(0x7f2ac41304f0 legacy=0x7f2ac4133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.479 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2aca43c700 10 monclient: _finish_hunting 0 2021-03-09T08:48:04.480 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2aca43c700 1 monclient: found mon.noname-c 2021-03-09T08:48:04.480 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2aca43c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:04.480 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2aca43c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.480 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2aca43c700 1 -- 172.21.15.195:0/1232625548 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac40a4730 con 0x7f2ac40a5fd0 2021-03-09T08:48:04.480 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2021-03-09T08:48:04.481 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2aca43c700 1 --2- 172.21.15.195:0/1232625548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac40a5fd0 0x7f2ac40a4240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f2ac40a48c0 tx=0x7f2ab8006e20).ready entity=mon.1 client_cookie=16415ddf61eed996 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:04.481 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 1 -- 172.21.15.195:0/1232625548 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab800beb0 con 0x7f2ac40a5fd0 2021-03-09T08:48:04.481 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.481 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T08:48:04.482 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 10 monclient: dump: 2021-03-09T08:48:04.482 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.482 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.482 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.482 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.483 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.483 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.483 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.483 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.483 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.484 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.484 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 10 monclient: _finish_auth 0 2021-03-09T08:48:04.484 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:04.484 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 5 monclient: authenticate success, global_id 4165 2021-03-09T08:48:04.485 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 1 -- 172.21.15.195:0/1232625548 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab8009760 con 0x7f2ac40a5fd0 2021-03-09T08:48:04.485 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:04.485 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:04.485 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:04.485 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 4 set_mon_vals no callback set 2021-03-09T08:48:04.486 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 1 -- 172.21.15.195:0/1232625548 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab8009ae0 con 0x7f2ac40a5fd0 2021-03-09T08:48:04.486 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.486 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:04.486 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2ac943a700 10 monclient: dump: 2021-03-09T08:48:04.487 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.488 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.488 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.489 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.489 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.489 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.490 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.490 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.490 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.490 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.491 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 10 monclient: shutdown 2021-03-09T08:48:04.492 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:04.492 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 1 -- 172.21.15.195:0/1232625548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac40a5fd0 msgr2=0x7f2ac40a4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.492 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 1 --2- 172.21.15.195:0/1232625548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac40a5fd0 0x7f2ac40a4240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f2ac40a48c0 tx=0x7f2ab8006e20).stop 2021-03-09T08:48:04.492 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 1 -- 172.21.15.195:0/1232625548 shutdown_connections 2021-03-09T08:48:04.493 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 1 --2- 172.21.15.195:0/1232625548 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac40a5fd0 0x7f2ac40a4240 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.493 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 1 -- 172.21.15.195:0/1232625548 >> 172.21.15.195:0/1232625548 conn(0x7f2ac40642c0 msgr2=0x7f2ac4069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:04.493 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 1 -- 172.21.15.195:0/1232625548 shutdown_connections 2021-03-09T08:48:04.493 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.454+0000 7f2acb43e700 1 -- 172.21.15.195:0/1232625548 wait complete. 2021-03-09T08:48:04.494 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.455+0000 7f2acb43e700 10 monclient: build_initial_monmap 2021-03-09T08:48:04.494 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.455+0000 7f2acb43e700 10 monclient: monmap: 2021-03-09T08:48:04.494 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 0 2021-03-09T08:48:04.494 INFO:teuthology.orchestra.run.smithi195.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:04.494 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 0.000000 2021-03-09T08:48:04.495 INFO:teuthology.orchestra.run.smithi195.stderr:created 0.000000 2021-03-09T08:48:04.495 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:04.495 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 1 2021-03-09T08:48:04.495 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:04.496 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:04.496 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:04.496 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.496 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.456+0000 7f2acb43e700 1 librados: starting msgr at 2021-03-09T08:48:04.496 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.456+0000 7f2acb43e700 1 librados: starting objecter 2021-03-09T08:48:04.497 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.456+0000 7f2acb43e700 1 Processor -- start 2021-03-09T08:48:04.497 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.456+0000 7f2acb43e700 1 -- start start 2021-03-09T08:48:04.497 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.456+0000 7f2acb43e700 1 librados: setting wanted keys 2021-03-09T08:48:04.497 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.456+0000 7f2acb43e700 1 librados: calling monclient init 2021-03-09T08:48:04.497 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.456+0000 7f2acb43e700 10 monclient: init 2021-03-09T08:48:04.498 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:04.498 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:48:04.498 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f2ac40a3950 0x7f2ac4139610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.498 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient(hunting): picked mon.noname-c con 0x7f2ac40a3950 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:04.499 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ac40a5fd0 0x7f2ac4137520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.499 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient(hunting): picked mon.noname-a con 0x7f2ac40a5fd0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:04.499 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac41304f0 0x7f2ac413ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:04.499 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient(hunting): picked mon.noname-b con 0x7f2ac41304f0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:04.499 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.500 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac4068320 con 0x7f2ac40a5fd0 2021-03-09T08:48:04.500 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.500 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac4139b30 con 0x7f2ac41304f0 2021-03-09T08:48:04.500 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:04.500 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ac4137a10 con 0x7f2ac40a3950 2021-03-09T08:48:04.501 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:04.501 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acb43e700 10 monclient(hunting): authenticate will time out at 849.947326s 2021-03-09T08:48:04.501 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac41304f0 0x7f2ac413ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.501 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac41304f0 0x7f2ac413ed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.195:45862/0 (socket says 172.21.15.195:45862) 2021-03-09T08:48:04.502 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 -- 172.21.15.195:0/3263101860 learned_addr learned my addr 172.21.15.195:0/3263101860 (peer_addr_for_me v2:172.21.15.195:0/0) 2021-03-09T08:48:04.502 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 10 monclient(hunting): get_auth_request con 0x7f2ac41304f0 auth_method 0 2021-03-09T08:48:04.502 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.502 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.502 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ac9c3b700 1 --2- 172.21.15.195:0/3263101860 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ac40a5fd0 0x7f2ac4137520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.503 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2aca43c700 1 --2- 172.21.15.195:0/3263101860 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f2ac40a3950 0x7f2ac4139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:04.503 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.503 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.503 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.504 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ac9c3b700 10 monclient(hunting): get_auth_request con 0x7f2ac40a5fd0 auth_method 0 2021-03-09T08:48:04.504 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ac9c3b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.504 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ac9c3b700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.504 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2aca43c700 10 monclient(hunting): get_auth_request con 0x7f2ac40a3950 auth_method 0 2021-03-09T08:48:04.504 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2aca43c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:04.505 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2aca43c700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:04.505 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ac9c3b700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.505 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ac9c3b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.505 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ac9c3b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.505 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2aca43c700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:04.506 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2aca43c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:04.506 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2aca43c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:04.506 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 10 monclient(hunting): handle_auth_done global_id 4171 payload 995 2021-03-09T08:48:04.506 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 -- 172.21.15.195:0/3263101860 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f2ac40a3950 msgr2=0x7f2ac4139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.507 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 --2- 172.21.15.195:0/3263101860 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f2ac40a3950 0x7f2ac4139610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.507 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 -- 172.21.15.195:0/3263101860 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ac40a5fd0 msgr2=0x7f2ac4137520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.507 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 --2- 172.21.15.195:0/3263101860 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ac40a5fd0 0x7f2ac4137520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.507 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 10 monclient: _finish_hunting 0 2021-03-09T08:48:04.507 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 monclient: found mon.noname-b 2021-03-09T08:48:04.508 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:04.508 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.508 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 -- 172.21.15.195:0/3263101860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac4136e00 con 0x7f2ac41304f0 2021-03-09T08:48:04.508 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2acac3d700 1 --2- 172.21.15.195:0/3263101860 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac41304f0 0x7f2ac413ed60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2ac000cab0 tx=0x7f2ac0007f20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:04.508 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ab27fc700 10 client.?.objecter ms_handle_connect 0x7f2ac41304f0 2021-03-09T08:48:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ab27fc700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ab27fc700 1 -- 172.21.15.195:0/3263101860 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000a590 con 0x7f2ac41304f0 2021-03-09T08:48:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ab27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.509 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ab27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ab27fc700 10 monclient: dump: 2021-03-09T08:48:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.510 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.511 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.511 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.511 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.511 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.511 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.512 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ab27fc700 10 monclient: _finish_auth 0 2021-03-09T08:48:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.457+0000 7f2ab27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:04.512 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 1 -- 172.21.15.195:0/3263101860 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac000a6f0 con 0x7f2ac41304f0 2021-03-09T08:48:04.513 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 5 monclient: authenticate success, global_id 4171 2021-03-09T08:48:04.513 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 10 monclient: _renew_subs 2021-03-09T08:48:04.513 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.513 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 1 -- 172.21.15.195:0/3263101860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ac4068700 con 0x7f2ac41304f0 2021-03-09T08:48:04.513 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:04.514 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 1 -- 172.21.15.195:0/3263101860 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac000aab0 con 0x7f2ac41304f0 2021-03-09T08:48:04.514 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:04.514 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:04.514 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 10 monclient: dump: 2021-03-09T08:48:04.514 INFO:teuthology.orchestra.run.smithi195.stderr:epoch 1 2021-03-09T08:48:04.515 INFO:teuthology.orchestra.run.smithi195.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:04.515 INFO:teuthology.orchestra.run.smithi195.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.515 INFO:teuthology.orchestra.run.smithi195.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:04.515 INFO:teuthology.orchestra.run.smithi195.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:04.516 INFO:teuthology.orchestra.run.smithi195.stderr:election_strategy: 3 2021-03-09T08:48:04.516 INFO:teuthology.orchestra.run.smithi195.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:04.516 INFO:teuthology.orchestra.run.smithi195.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:04.516 INFO:teuthology.orchestra.run.smithi195.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:04.516 INFO:teuthology.orchestra.run.smithi195.stderr: 2021-03-09T08:48:04.517 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ac8c39700 4 set_mon_vals no callback set 2021-03-09T08:48:04.517 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 10 client.4171.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:04.517 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 10 monclient: _renew_subs 2021-03-09T08:48:04.517 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.517 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 1 -- 172.21.15.195:0/3263101860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ac4067f50 con 0x7f2ac41304f0 2021-03-09T08:48:04.518 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 1 librados: init done 2021-03-09T08:48:04.518 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 1 -- 172.21.15.195:0/3263101860 <== mon.1 v2:172.21.15.195:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7f2ac0019030 con 0x7f2ac41304f0 2021-03-09T08:48:04.518 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 1 -- 172.21.15.195:0/3263101860 <== mon.1 v2:172.21.15.195:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7f2ac0003030 con 0x7f2ac41304f0 2021-03-09T08:48:04.518 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 10 client.4171.objecter ms_dispatch 0x7f2ac4001550 osd_map(9..9 src has 1..9) v4 2021-03-09T08:48:04.519 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 3 client.4171.objecter handle_osd_map got epochs [9,9] > 0 2021-03-09T08:48:04.519 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 3 client.4171.objecter handle_osd_map decoding full epoch 9 2021-03-09T08:48:04.519 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2ab27fc700 20 client.4171.objecter dump_active .. 0 homeless 2021-03-09T08:48:04.519 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:04.519 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:04.520 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.520 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.458+0000 7f2acb43e700 1 -- 172.21.15.195:0/3263101860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ac40a7480 con 0x7f2ac41304f0 2021-03-09T08:48:04.520 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.460+0000 7f2ab27fc700 1 -- 172.21.15.195:0/3263101860 <== mon.1 v2:172.21.15.195:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7f2ac0021060 con 0x7f2ac41304f0 2021-03-09T08:48:04.520 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.460+0000 7f2ab27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:04.520 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.460+0000 7f2ab27fc700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:04.541 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.539+0000 7f2acb43e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b11e00a8-aaeb-49b0-89a4-9720edf16c13", "id": 7}] 2021-03-09T08:48:04.541 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.539+0000 7f2acb43e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b11e00a8-aaeb-49b0-89a4-9720edf16c13", "id": 7}] 2021-03-09T08:48:04.541 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.539+0000 7f2acb43e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.195:3300/0 2021-03-09T08:48:04.542 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.539+0000 7f2acb43e700 1 -- 172.21.15.195:0/3263101860 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b11e00a8-aaeb-49b0-89a4-9720edf16c13", "id": 7} v 0) v1 -- 0x7f2ac40589f0 con 0x7f2ac41304f0 2021-03-09T08:48:04.544 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.542+0000 7f2ab27fc700 1 -- 172.21.15.195:0/3263101860 <== mon.1 v2:172.21.15.195:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b11e00a8-aaeb-49b0-89a4-9720edf16c13", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2ac0017070 con 0x7f2ac41304f0 2021-03-09T08:48:04.544 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.542+0000 7f2ab27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b11e00a8-aaeb-49b0-89a4-9720edf16c13", "id": 7}] 2021-03-09T08:48:04.544 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.542+0000 7f2ab27fc700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:04.545 INFO:teuthology.orchestra.run.smithi195.stdout:7 2021-03-09T08:48:04.546 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 10 librados: watch_flush enter 2021-03-09T08:48:04.547 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 10 librados: watch_flush exit 2021-03-09T08:48:04.547 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 20 client.4171.objecter shutdown clearing up homeless session... 2021-03-09T08:48:04.547 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 10 client.4171.objecter successfully canceled tick 2021-03-09T08:48:04.547 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 10 monclient: shutdown 2021-03-09T08:48:04.547 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:04.548 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 -- 172.21.15.195:0/3263101860 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac41304f0 msgr2=0x7f2ac413ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:04.548 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 --2- 172.21.15.195:0/3263101860 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac41304f0 0x7f2ac413ed60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2ac000cab0 tx=0x7f2ac0007f20).stop 2021-03-09T08:48:04.548 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 -- 172.21.15.195:0/3263101860 shutdown_connections 2021-03-09T08:48:04.548 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 --2- 172.21.15.195:0/3263101860 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f2ac41304f0 0x7f2ac413ed60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.549 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 --2- 172.21.15.195:0/3263101860 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f2ac40a3950 0x7f2ac4139610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.549 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 --2- 172.21.15.195:0/3263101860 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f2ac40a5fd0 0x7f2ac4137520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:04.549 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 -- 172.21.15.195:0/3263101860 >> 172.21.15.195:0/3263101860 conn(0x7f2ac40642c0 msgr2=0x7f2ac412fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:04.549 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 -- 172.21.15.195:0/3263101860 shutdown_connections 2021-03-09T08:48:04.549 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 -- 172.21.15.195:0/3263101860 wait complete. 2021-03-09T08:48:04.550 INFO:teuthology.orchestra.run.smithi195.stderr:2021-03-09T08:48:04.543+0000 7f2acb43e700 1 librados: shutdown 2021-03-09T08:48:04.560 INFO:tasks.ceph.osd.0:Restarting daemon 2021-03-09T08:48:04.560 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-03-09T08:48:04.567 INFO:tasks.ceph.osd.0:Started 2021-03-09T08:48:04.567 INFO:tasks.ceph.osd.1:Restarting daemon 2021-03-09T08:48:04.568 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-03-09T08:48:04.577 INFO:tasks.ceph.osd.1:Started 2021-03-09T08:48:04.577 INFO:tasks.ceph.osd.2:Restarting daemon 2021-03-09T08:48:04.578 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-03-09T08:48:04.583 INFO:tasks.ceph.osd.2:Started 2021-03-09T08:48:04.583 INFO:tasks.ceph.osd.3:Restarting daemon 2021-03-09T08:48:04.584 DEBUG:teuthology.orchestra.run.smithi099:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2021-03-09T08:48:04.594 INFO:tasks.ceph.osd.3:Started 2021-03-09T08:48:04.594 INFO:tasks.ceph.osd.4:Restarting daemon 2021-03-09T08:48:04.595 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2021-03-09T08:48:04.601 INFO:tasks.ceph.osd.4:Started 2021-03-09T08:48:04.602 INFO:tasks.ceph.osd.5:Restarting daemon 2021-03-09T08:48:04.602 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2021-03-09T08:48:04.606 INFO:tasks.ceph.osd.5:Started 2021-03-09T08:48:04.606 INFO:tasks.ceph.osd.6:Restarting daemon 2021-03-09T08:48:04.607 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2021-03-09T08:48:04.611 INFO:tasks.ceph.osd.6:Started 2021-03-09T08:48:04.611 INFO:tasks.ceph.osd.7:Restarting daemon 2021-03-09T08:48:04.612 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2021-03-09T08:48:04.616 INFO:tasks.ceph.osd.7:Started 2021-03-09T08:48:04.617 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-09T08:48:05.181 INFO:tasks.ceph.osd.0.smithi099.stderr:2021-03-09T08:48:05.179+0000 7f74ec5fe080 -1 Falling back to public interface 2021-03-09T08:48:05.217 INFO:tasks.ceph.osd.1.smithi099.stderr:2021-03-09T08:48:05.215+0000 7fefb9b98080 -1 Falling back to public interface 2021-03-09T08:48:05.218 INFO:tasks.ceph.osd.2.smithi099.stderr:2021-03-09T08:48:05.216+0000 7ff3bd616080 -1 Falling back to public interface 2021-03-09T08:48:05.221 INFO:tasks.ceph.osd.4.smithi195.stderr:2021-03-09T08:48:05.219+0000 7f6b0e107080 -1 Falling back to public interface 2021-03-09T08:48:05.221 INFO:tasks.ceph.osd.3.smithi099.stderr:2021-03-09T08:48:05.219+0000 7f69f6561080 -1 Falling back to public interface 2021-03-09T08:48:05.222 INFO:tasks.ceph.osd.5.smithi195.stderr:2021-03-09T08:48:05.220+0000 7f7f08da9080 -1 Falling back to public interface 2021-03-09T08:48:05.231 INFO:tasks.ceph.osd.7.smithi195.stderr:2021-03-09T08:48:05.229+0000 7f09c5437080 -1 Falling back to public interface 2021-03-09T08:48:05.242 INFO:tasks.ceph.osd.6.smithi195.stderr:2021-03-09T08:48:05.240+0000 7efd21fb7080 -1 Falling back to public interface 2021-03-09T08:48:05.422 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T08:48:05.553 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.550+0000 7fc01a2e9700 10 monclient: get_monmap_and_config 2021-03-09T08:48:05.553 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.550+0000 7fc01a2e9700 10 monclient: build_initial_monmap 2021-03-09T08:48:05.553 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.550+0000 7fc01a2e9700 10 monclient: monmap: 2021-03-09T08:48:05.554 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2021-03-09T08:48:05.554 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:05.554 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:48:05.552111+0000 2021-03-09T08:48:05.554 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:48:05.552111+0000 2021-03-09T08:48:05.554 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:05.555 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:05.555 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:05.555 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:05.555 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:05.556 INFO:teuthology.misc.health.smithi099.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:05.556 INFO:teuthology.misc.health.smithi099.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:05.556 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:05.557 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.551+0000 7fc01a2e9700 1 Processor -- start 2021-03-09T08:48:05.557 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.551+0000 7fc01a2e9700 1 -- start start 2021-03-09T08:48:05.558 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.551+0000 7fc01a2e9700 10 monclient: init 2021-03-09T08:48:05.558 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:05.558 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2021-03-09T08:48:05.558 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0140a61d0 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:05.559 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc014065320 0x7fc0140673e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:05.559 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 10 monclient(hunting): picked mon.noname-a con 0x7fc014065320 addr v2:172.21.15.99:3300/0 2021-03-09T08:48:05.559 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a2790 0x7fc0140a2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:05.559 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 10 monclient(hunting): picked mon.noname-b con 0x7fc0140a2790 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:05.559 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0140a65b0 con 0x7fc0140a61d0 2021-03-09T08:48:05.560 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:05.560 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc014066d50 con 0x7fc014065320 2021-03-09T08:48:05.570 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:05.570 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0140a3110 con 0x7fc0140a2790 2021-03-09T08:48:05.570 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:05.571 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc01a2e9700 10 monclient(hunting): authenticate will time out at 852.027291s 2021-03-09T08:48:05.571 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc014065320 0x7fc0140673e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:05.571 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a2790 0x7fc0140a2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:05.571 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a2790 0x7fc0140a2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.99:3301/0 says I am v2:172.21.15.99:56576/0 (socket says 172.21.15.99:56576) 2021-03-09T08:48:05.572 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7fc014065320 0x7fc0140673e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33822/0 (socket says 172.21.15.99:33822) 2021-03-09T08:48:05.572 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 1 -- 172.21.15.99:0/2315597476 learned_addr learned my addr 172.21.15.99:0/2315597476 (peer_addr_for_me v2:172.21.15.99:0/0) 2021-03-09T08:48:05.572 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 10 monclient(hunting): get_auth_request con 0x7fc0140a2790 auth_method 0 2021-03-09T08:48:05.572 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:05.572 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:05.573 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 10 monclient(hunting): get_auth_request con 0x7fc014065320 auth_method 0 2021-03-09T08:48:05.573 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:05.573 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:05.573 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:05.574 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:05.574 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:05.574 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:05.574 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:05.574 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:05.575 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 10 monclient(hunting): handle_auth_done global_id 4173 payload 450 2021-03-09T08:48:05.575 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 1 -- 172.21.15.99:0/2315597476 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a2790 msgr2=0x7fc0140a2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2021-03-09T08:48:05.575 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 1 --2- 172.21.15.99:0/2315597476 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a2790 0x7fc0140a2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:05.575 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 1 -- 172.21.15.99:0/2315597476 >> v1:172.21.15.195:6789/0 conn(0x7fc0140a61d0 legacy=0x7fc014136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:05.576 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 10 monclient: _finish_hunting 0 2021-03-09T08:48:05.576 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 1 monclient: found mon.noname-a 2021-03-09T08:48:05.576 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:05.576 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.99:3300/0 2021-03-09T08:48:05.577 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 1 -- 172.21.15.99:0/2315597476 --> v2:172.21.15.99:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014066f40 con 0x7fc014065320 2021-03-09T08:48:05.577 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc018886700 1 --2- 172.21.15.99:0/2315597476 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a2790 0x7fc0140a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2021-03-09T08:48:05.577 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc0137fe700 1 --2- 172.21.15.99:0/2315597476 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 0x7fc0140673e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fc000006fd0 tx=0x7fc000006eb0).ready entity=mon.0 client_cookie=30ed1991789f8e0b server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:05.577 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.552+0000 7fc012ffd700 1 -- 172.21.15.99:0/2315597476 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000a740 con 0x7fc014065320 2021-03-09T08:48:05.577 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:05.578 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-03-09T08:48:05.578 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 10 monclient: dump: 2021-03-09T08:48:05.578 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2021-03-09T08:48:05.578 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:05.578 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:05.579 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:05.579 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:05.579 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:05.579 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:05.580 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:05.580 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:05.580 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:05.580 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 10 monclient: _finish_auth 0 2021-03-09T08:48:05.580 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:05.581 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 5 monclient: authenticate success, global_id 4173 2021-03-09T08:48:05.581 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:05.581 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 1 -- 172.21.15.99:0/2315597476 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00000a8a0 con 0x7fc014065320 2021-03-09T08:48:05.581 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:05.582 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:05.582 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 4 set_mon_vals no callback set 2021-03-09T08:48:05.582 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 1 -- 172.21.15.99:0/2315597476 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000ac20 con 0x7fc014065320 2021-03-09T08:48:05.582 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:05.583 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-03-09T08:48:05.583 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc012ffd700 10 monclient: dump: 2021-03-09T08:48:05.583 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2021-03-09T08:48:05.583 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:05.583 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:05.584 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:05.584 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:05.584 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:05.584 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:05.584 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:05.585 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:05.585 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:05.585 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 10 monclient: shutdown 2021-03-09T08:48:05.585 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:05.585 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 1 -- 172.21.15.99:0/2315597476 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 msgr2=0x7fc0140673e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:05.586 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 1 --2- 172.21.15.99:0/2315597476 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 0x7fc0140673e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fc000006fd0 tx=0x7fc000006eb0).stop 2021-03-09T08:48:05.586 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 1 -- 172.21.15.99:0/2315597476 shutdown_connections 2021-03-09T08:48:05.586 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 1 --2- 172.21.15.99:0/2315597476 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a2790 0x7fc0140a2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:05.586 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 1 --2- 172.21.15.99:0/2315597476 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 0x7fc0140673e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:05.587 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 1 -- 172.21.15.99:0/2315597476 >> 172.21.15.99:0/2315597476 conn(0x7fc014069f90 msgr2=0x7fc01406a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:05.587 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 1 -- 172.21.15.99:0/2315597476 shutdown_connections 2021-03-09T08:48:05.587 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.553+0000 7fc01a2e9700 1 -- 172.21.15.99:0/2315597476 wait complete. 2021-03-09T08:48:05.587 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.554+0000 7fc01a2e9700 10 monclient: build_initial_monmap 2021-03-09T08:48:05.588 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.554+0000 7fc01a2e9700 10 monclient: monmap: 2021-03-09T08:48:05.588 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2021-03-09T08:48:05.588 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:05.588 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2021-03-09T08:48:05.589 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2021-03-09T08:48:05.589 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:05.589 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2021-03-09T08:48:05.589 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:05.589 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:05.590 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:05.590 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:05.590 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.554+0000 7fc01a2e9700 1 librados: starting msgr at 2021-03-09T08:48:05.590 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.554+0000 7fc01a2e9700 1 librados: starting objecter 2021-03-09T08:48:05.591 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.554+0000 7fc01a2e9700 1 Processor -- start 2021-03-09T08:48:05.591 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.554+0000 7fc01a2e9700 1 -- start start 2021-03-09T08:48:05.591 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.554+0000 7fc01a2e9700 1 librados: setting wanted keys 2021-03-09T08:48:05.591 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.554+0000 7fc01a2e9700 1 librados: calling monclient init 2021-03-09T08:48:05.592 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.554+0000 7fc01a2e9700 10 monclient: init 2021-03-09T08:48:05.592 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:05.592 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:48:05.592 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 0x7fc014114ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:05.592 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient(hunting): picked mon.noname-a con 0x7fc014065320 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:05.593 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0140a2790 0x7fc0140a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:05.593 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient(hunting): picked mon.noname-b con 0x7fc0140a2790 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:05.593 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a61d0 0x7fc014145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:05.593 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient(hunting): picked mon.noname-c con 0x7fc0140a61d0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:05.593 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:05.594 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0141458d0 con 0x7fc014065320 2021-03-09T08:48:05.594 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:05.594 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc014115190 con 0x7fc0140a2790 2021-03-09T08:48:05.594 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:05.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0140a1700 con 0x7fc0140a61d0 2021-03-09T08:48:05.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:05.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc01a2e9700 10 monclient(hunting): authenticate will time out at 852.030375s 2021-03-09T08:48:05.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 0x7fc014114ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:05.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 0x7fc014114ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.99:3300/0 says I am v2:172.21.15.99:33828/0 (socket says 172.21.15.99:33828) 2021-03-09T08:48:05.596 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 -- 172.21.15.99:0/3517038055 learned_addr learned my addr 172.21.15.99:0/3517038055 (peer_addr_for_me v2:172.21.15.99:0/0) 2021-03-09T08:48:05.596 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 10 monclient(hunting): get_auth_request con 0x7fc014065320 auth_method 0 2021-03-09T08:48:05.596 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:05.596 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:05.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc018886700 1 --2- 172.21.15.99:0/3517038055 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a61d0 0x7fc014145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:05.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:05.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:05.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc0137fe700 1 --2- 172.21.15.99:0/3517038055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0140a2790 0x7fc0140a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:05.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:05.598 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc018886700 10 monclient(hunting): get_auth_request con 0x7fc0140a61d0 auth_method 0 2021-03-09T08:48:05.598 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc018886700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:05.598 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc018886700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:05.598 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc0137fe700 10 monclient(hunting): get_auth_request con 0x7fc0140a2790 auth_method 0 2021-03-09T08:48:05.599 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc0137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:05.599 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc0137fe700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:05.599 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc018886700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:05.599 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc018886700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:05.599 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc018886700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:05.600 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc0137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:05.600 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc0137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:05.600 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc0137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:05.600 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 10 monclient(hunting): handle_auth_done global_id 4176 payload 995 2021-03-09T08:48:05.600 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 -- 172.21.15.99:0/3517038055 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a61d0 msgr2=0x7fc014145d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:05.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 --2- 172.21.15.99:0/3517038055 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a61d0 0x7fc014145d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:05.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 -- 172.21.15.99:0/3517038055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0140a2790 msgr2=0x7fc0140a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:05.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 --2- 172.21.15.99:0/3517038055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0140a2790 0x7fc0140a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:05.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 10 monclient: _finish_hunting 0 2021-03-09T08:48:05.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 monclient: found mon.noname-a 2021-03-09T08:48:05.602 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:05.602 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.99:3300/0 2021-03-09T08:48:05.602 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 -- 172.21.15.99:0/3517038055 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014146260 con 0x7fc014065320 2021-03-09T08:48:05.602 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fc013fff700 1 --2- 172.21.15.99:0/3517038055 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 0x7fc014114ca0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc00400c710 tx=0x7fc004007c80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:05.603 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fbffffff700 10 client.?.objecter ms_handle_connect 0x7fc014065320 2021-03-09T08:48:05.603 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.555+0000 7fbffffff700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:05.603 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 1 -- 172.21.15.99:0/3517038055 <== mon.0 v2:172.21.15.99:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004001ea0 con 0x7fc014065320 2021-03-09T08:48:05.603 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:05.603 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2021-03-09T08:48:05.604 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 10 monclient: dump: 2021-03-09T08:48:05.604 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2021-03-09T08:48:05.604 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:05.604 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:05.604 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:05.605 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:05.605 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:05.605 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:05.605 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:05.605 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:05.606 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:05.606 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 10 monclient: _finish_auth 0 2021-03-09T08:48:05.606 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:05.606 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 5 monclient: authenticate success, global_id 4176 2021-03-09T08:48:05.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 1 -- 172.21.15.99:0/3517038055 <== mon.0 v2:172.21.15.99:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc004007020 con 0x7fc014065320 2021-03-09T08:48:05.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 10 monclient: _renew_subs 2021-03-09T08:48:05.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.99:3300/0 2021-03-09T08:48:05.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 1 -- 172.21.15.99:0/3517038055 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc014145470 con 0x7fc014065320 2021-03-09T08:48:05.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:05.608 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 1 -- 172.21.15.99:0/3517038055 <== mon.0 v2:172.21.15.99:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0040073c0 con 0x7fc014065320 2021-03-09T08:48:05.608 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:05.608 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2021-03-09T08:48:05.608 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 10 monclient: dump: 2021-03-09T08:48:05.608 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2021-03-09T08:48:05.609 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:05.609 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:05.609 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:05.609 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:05.609 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:05.610 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:05.610 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:05.610 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:05.610 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:05.610 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 10 client.4176.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:05.611 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 10 monclient: _renew_subs 2021-03-09T08:48:05.611 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.99:3300/0 2021-03-09T08:48:05.611 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 1 -- 172.21.15.99:0/3517038055 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc014110af0 con 0x7fc014065320 2021-03-09T08:48:05.611 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc0127fc700 4 set_mon_vals no callback set 2021-03-09T08:48:05.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 1 librados: init done 2021-03-09T08:48:05.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 1 -- 172.21.15.99:0/3517038055 <== mon.0 v2:172.21.15.99:3300/0 4 ==== mgrmap(e 1) v1 ==== 623+0+0 (secure 0 0 0) 0x7fc00401a060 con 0x7fc014065320 2021-03-09T08:48:05.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 1 -- 172.21.15.99:0/3517038055 <== mon.0 v2:172.21.15.99:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7fc004013d40 con 0x7fc014065320 2021-03-09T08:48:05.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 10 client.4176.objecter ms_dispatch 0x7fc014001550 osd_map(10..10 src has 1..10) v4 2021-03-09T08:48:05.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 3 client.4176.objecter handle_osd_map got epochs [10,10] > 0 2021-03-09T08:48:05.613 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 3 client.4176.objecter handle_osd_map decoding full epoch 10 2021-03-09T08:48:05.613 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fbffffff700 20 client.4176.objecter dump_active .. 0 homeless 2021-03-09T08:48:05.613 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:05.613 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:05.613 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.556+0000 7fc01a2e9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.99:3300/0 2021-03-09T08:48:05.614 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.557+0000 7fc01a2e9700 1 -- 172.21.15.99:0/3517038055 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0140589f0 con 0x7fc014065320 2021-03-09T08:48:05.614 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.559+0000 7fbffffff700 1 -- 172.21.15.99:0/3517038055 <== mon.0 v2:172.21.15.99:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72044 (secure 0 0 0) 0x7fc004018070 con 0x7fc014065320 2021-03-09T08:48:05.614 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.559+0000 7fbffffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:05.614 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.559+0000 7fbffffff700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:05.627 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.624+0000 7fc01a2e9700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-03-09T08:48:05.627 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.624+0000 7fc01a2e9700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-03-09T08:48:05.627 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.624+0000 7fc01a2e9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.99:3300/0 2021-03-09T08:48:05.627 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.624+0000 7fc01a2e9700 1 -- 172.21.15.99:0/3517038055 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc014065900 con 0x7fc014065320 2021-03-09T08:48:05.628 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.624+0000 7fbffffff700 1 -- 172.21.15.99:0/3517038055 <== mon.0 v2:172.21.15.99:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7023 (secure 0 0 0) 0x7fc0140a65b0 con 0x7fc014065320 2021-03-09T08:48:05.628 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.624+0000 7fbffffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-03-09T08:48:05.628 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.624+0000 7fbffffff700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:05.629 INFO:teuthology.misc.health.smithi099.stdout: 2021-03-09T08:48:05.629 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":10,"fsid":"e7417d55-b73a-46ac-8f0e-ad6e583f612d","created":"2021-03-09T08:48:02.065911+0000","modified":"2021-03-09T08:48:04.540876+0000","last_up_change":"0.000000","last_in_change":"2021-03-09T08:48:04.540876+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"bfcce70f-6a3c-4d76-9a45-145c7d5e468d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ff4574a3-9580-4767-b99d-f2d4e488b465","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb8b7997-db54-48ba-8336-f8566c57204f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc1a81b3-b2c9-4c37-9429-6d131c9e791c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"802be6f1-4e15-4e2d-8720-0a3009f8d02e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"17c2bea4-c1fc-49d2-a191-957e9575fd9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"27bb6c24-657c-4887-b344-aa7c2dd5676f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b11e00a8-aaeb-49b0-89a4-9720edf16c13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-09T08:48:05.630 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 10 librados: watch_flush enter 2021-03-09T08:48:05.631 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 10 librados: watch_flush exit 2021-03-09T08:48:05.631 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 20 client.4176.objecter shutdown clearing up homeless session... 2021-03-09T08:48:05.631 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 10 client.4176.objecter successfully canceled tick 2021-03-09T08:48:05.631 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 10 monclient: shutdown 2021-03-09T08:48:05.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:05.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 -- 172.21.15.99:0/3517038055 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 msgr2=0x7fc014114ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:05.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 --2- 172.21.15.99:0/3517038055 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 0x7fc014114ca0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fc00400c710 tx=0x7fc004007c80).stop 2021-03-09T08:48:05.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 -- 172.21.15.99:0/3517038055 shutdown_connections 2021-03-09T08:48:05.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 --2- 172.21.15.99:0/3517038055 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7fc0140a61d0 0x7fc014145d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:05.633 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 --2- 172.21.15.99:0/3517038055 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7fc014065320 0x7fc014114ca0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:05.633 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 --2- 172.21.15.99:0/3517038055 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7fc0140a2790 0x7fc0140a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:05.633 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 -- 172.21.15.99:0/3517038055 >> 172.21.15.99:0/3517038055 conn(0x7fc014069f90 msgr2=0x7fc01406a370 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:05.633 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 -- 172.21.15.99:0/3517038055 shutdown_connections 2021-03-09T08:48:05.634 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 -- 172.21.15.99:0/3517038055 wait complete. 2021-03-09T08:48:05.634 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:05.625+0000 7fc01a2e9700 1 librados: shutdown 2021-03-09T08:48:05.638 DEBUG:teuthology.misc:0 of 8 OSDs are up 2021-03-09T08:48:08.642 INFO:tasks.ceph.osd.0.smithi099.stderr:2021-03-09T08:48:08.640+0000 7f74ec5fe080 -1 osd.0 0 log_to_monitors {default=true} 2021-03-09T08:48:08.681 INFO:tasks.ceph.osd.4.smithi195.stderr:2021-03-09T08:48:08.679+0000 7f6b0e107080 -1 osd.4 0 log_to_monitors {default=true} 2021-03-09T08:48:08.686 INFO:tasks.ceph.osd.2.smithi099.stderr:2021-03-09T08:48:08.684+0000 7ff3bd616080 -1 osd.2 0 log_to_monitors {default=true} 2021-03-09T08:48:08.697 INFO:tasks.ceph.osd.3.smithi099.stderr:2021-03-09T08:48:08.695+0000 7f69f6561080 -1 osd.3 0 log_to_monitors {default=true} 2021-03-09T08:48:08.697 INFO:tasks.ceph.osd.1.smithi099.stderr:2021-03-09T08:48:08.695+0000 7fefb9b98080 -1 osd.1 0 log_to_monitors {default=true} 2021-03-09T08:48:08.700 INFO:tasks.ceph.osd.7.smithi195.stderr:2021-03-09T08:48:08.698+0000 7f09c5437080 -1 osd.7 0 log_to_monitors {default=true} 2021-03-09T08:48:08.712 INFO:tasks.ceph.osd.5.smithi195.stderr:2021-03-09T08:48:08.711+0000 7f7f08da9080 -1 osd.5 0 log_to_monitors {default=true} 2021-03-09T08:48:08.720 INFO:tasks.ceph.osd.6.smithi195.stderr:2021-03-09T08:48:08.719+0000 7efd21fb7080 -1 osd.6 0 log_to_monitors {default=true} 2021-03-09T08:48:10.157 INFO:tasks.ceph.osd.4.smithi195.stderr:2021-03-09T08:48:10.154+0000 7f6b04884700 -1 osd.4 0 waiting for initial osdmap 2021-03-09T08:48:10.159 INFO:tasks.ceph.osd.0.smithi099.stderr:2021-03-09T08:48:10.156+0000 7f74e2d7b700 -1 osd.0 0 waiting for initial osdmap 2021-03-09T08:48:10.161 INFO:tasks.ceph.osd.1.smithi099.stderr:2021-03-09T08:48:10.158+0000 7fefb0315700 -1 osd.1 0 waiting for initial osdmap 2021-03-09T08:48:10.161 INFO:tasks.ceph.osd.5.smithi195.stderr:2021-03-09T08:48:10.159+0000 7f7eff526700 -1 osd.5 0 waiting for initial osdmap 2021-03-09T08:48:10.162 INFO:tasks.ceph.osd.3.smithi099.stderr:2021-03-09T08:48:10.159+0000 7f69eccde700 -1 osd.3 0 waiting for initial osdmap 2021-03-09T08:48:10.162 INFO:tasks.ceph.osd.6.smithi195.stderr:2021-03-09T08:48:10.159+0000 7efd18734700 -1 osd.6 0 waiting for initial osdmap 2021-03-09T08:48:10.163 INFO:tasks.ceph.osd.2.smithi099.stderr:2021-03-09T08:48:10.161+0000 7ff3b3d93700 -1 osd.2 0 waiting for initial osdmap 2021-03-09T08:48:10.164 INFO:tasks.ceph.osd.7.smithi195.stderr:2021-03-09T08:48:10.161+0000 7f09bbbb4700 -1 osd.7 0 waiting for initial osdmap 2021-03-09T08:48:11.195 INFO:tasks.ceph.mon.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f97590a5700 time 2021-03-09T08:48:11.194218+0000 2021-03-09T08:48:11.195 INFO:tasks.ceph.mon.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:48:11.196 INFO:tasks.ceph.mon.a.smithi099.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:11.197 INFO:tasks.ceph.mon.a.smithi099.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f9766a171d8] 2021-03-09T08:48:11.197 INFO:tasks.ceph.mon.a.smithi099.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9766a173f2] 2021-03-09T08:48:11.197 INFO:tasks.ceph.mon.a.smithi099.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x562ff8fb697c] 2021-03-09T08:48:11.197 INFO:tasks.ceph.mon.a.smithi099.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x562ff8ff5ced] 2021-03-09T08:48:11.197 INFO:tasks.ceph.mon.a.smithi099.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x562ff901db5a] 2021-03-09T08:48:11.198 INFO:tasks.ceph.mon.a.smithi099.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x562ff9029054] 2021-03-09T08:48:11.198 INFO:tasks.ceph.mon.a.smithi099.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x562ff902d2c3] 2021-03-09T08:48:11.198 INFO:tasks.ceph.mon.a.smithi099.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x562ff8faec2d] 2021-03-09T08:48:11.198 INFO:tasks.ceph.mon.a.smithi099.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x562ff8e97c74] 2021-03-09T08:48:11.199 INFO:tasks.ceph.mon.a.smithi099.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x562ff8e9c4d9] 2021-03-09T08:48:11.199 INFO:tasks.ceph.mon.a.smithi099.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x562ff8e9d766] 2021-03-09T08:48:11.199 INFO:tasks.ceph.mon.a.smithi099.stderr: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x562ff8ecbb0c] 2021-03-09T08:48:11.199 INFO:tasks.ceph.mon.a.smithi099.stderr: 13: (DispatchQueue::entry()+0x126a) [0x7f9766c510ea] 2021-03-09T08:48:11.200 INFO:tasks.ceph.mon.a.smithi099.stderr: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9766cfeae1] 2021-03-09T08:48:11.200 INFO:tasks.ceph.mon.a.smithi099.stderr: 15: (Thread::_entry_func(void*)+0xd) [0x7f9766afe41d] 2021-03-09T08:48:11.200 INFO:tasks.ceph.mon.a.smithi099.stderr: 16: /lib64/libpthread.so.0(+0x814a) [0x7f97644fd14a] 2021-03-09T08:48:11.200 INFO:tasks.ceph.mon.a.smithi099.stderr: 17: clone() 2021-03-09T08:48:11.200 INFO:tasks.ceph.mon.a.smithi099.stderr:2021-03-09T08:48:11.194+0000 7f97590a5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f97590a5700 time 2021-03-09T08:48:11.194218+0000 2021-03-09T08:48:11.201 INFO:tasks.ceph.mon.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:48:11.201 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.201 INFO:tasks.ceph.mon.a.smithi099.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:11.201 INFO:tasks.ceph.mon.a.smithi099.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f9766a171d8] 2021-03-09T08:48:11.202 INFO:tasks.ceph.mon.a.smithi099.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9766a173f2] 2021-03-09T08:48:11.202 INFO:tasks.ceph.mon.a.smithi099.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x562ff8fb697c] 2021-03-09T08:48:11.202 INFO:tasks.ceph.mon.a.smithi099.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x562ff8ff5ced] 2021-03-09T08:48:11.202 INFO:tasks.ceph.mon.a.smithi099.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x562ff901db5a] 2021-03-09T08:48:11.203 INFO:tasks.ceph.mon.a.smithi099.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x562ff9029054] 2021-03-09T08:48:11.203 INFO:tasks.ceph.mon.a.smithi099.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x562ff902d2c3] 2021-03-09T08:48:11.203 INFO:tasks.ceph.mon.a.smithi099.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x562ff8faec2d] 2021-03-09T08:48:11.203 INFO:tasks.ceph.mon.a.smithi099.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x562ff8e97c74] 2021-03-09T08:48:11.204 INFO:tasks.ceph.mon.a.smithi099.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x562ff8e9c4d9] 2021-03-09T08:48:11.204 INFO:tasks.ceph.mon.a.smithi099.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x562ff8e9d766] 2021-03-09T08:48:11.204 INFO:tasks.ceph.mon.a.smithi099.stderr: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x562ff8ecbb0c] 2021-03-09T08:48:11.204 INFO:tasks.ceph.mon.a.smithi099.stderr: 13: (DispatchQueue::entry()+0x126a) [0x7f9766c510ea] 2021-03-09T08:48:11.204 INFO:tasks.ceph.mon.a.smithi099.stderr: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9766cfeae1] 2021-03-09T08:48:11.205 INFO:tasks.ceph.mon.a.smithi099.stderr: 15: (Thread::_entry_func(void*)+0xd) [0x7f9766afe41d] 2021-03-09T08:48:11.205 INFO:tasks.ceph.mon.a.smithi099.stderr: 16: /lib64/libpthread.so.0(+0x814a) [0x7f97644fd14a] 2021-03-09T08:48:11.205 INFO:tasks.ceph.mon.a.smithi099.stderr: 17: clone() 2021-03-09T08:48:11.205 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.205 INFO:tasks.ceph.mon.a.smithi099.stderr:*** Caught signal (Aborted) ** 2021-03-09T08:48:11.206 INFO:tasks.ceph.mon.a.smithi099.stderr: in thread 7f97590a5700 thread_name:ms_dispatch 2021-03-09T08:48:11.206 INFO:tasks.ceph.mon.a.smithi099.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:11.207 INFO:tasks.ceph.mon.a.smithi099.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9764507b20] 2021-03-09T08:48:11.207 INFO:tasks.ceph.mon.a.smithi099.stderr: 2: gsignal() 2021-03-09T08:48:11.207 INFO:tasks.ceph.mon.a.smithi099.stderr: 3: abort() 2021-03-09T08:48:11.207 INFO:tasks.ceph.mon.a.smithi099.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f9766a17229] 2021-03-09T08:48:11.208 INFO:tasks.ceph.mon.a.smithi099.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9766a173f2] 2021-03-09T08:48:11.208 INFO:tasks.ceph.mon.a.smithi099.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x562ff8fb697c] 2021-03-09T08:48:11.208 INFO:tasks.ceph.mon.a.smithi099.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x562ff8ff5ced] 2021-03-09T08:48:11.208 INFO:tasks.ceph.mon.a.smithi099.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x562ff901db5a] 2021-03-09T08:48:11.208 INFO:tasks.ceph.mon.a.smithi099.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x562ff9029054] 2021-03-09T08:48:11.209 INFO:tasks.ceph.mon.a.smithi099.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x562ff902d2c3] 2021-03-09T08:48:11.209 INFO:tasks.ceph.mon.a.smithi099.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x562ff8faec2d] 2021-03-09T08:48:11.209 INFO:tasks.ceph.mon.a.smithi099.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x562ff8e97c74] 2021-03-09T08:48:11.209 INFO:tasks.ceph.mon.a.smithi099.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x562ff8e9c4d9] 2021-03-09T08:48:11.209 INFO:tasks.ceph.mon.a.smithi099.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x562ff8e9d766] 2021-03-09T08:48:11.210 INFO:tasks.ceph.mon.a.smithi099.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x562ff8ecbb0c] 2021-03-09T08:48:11.210 INFO:tasks.ceph.mon.a.smithi099.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f9766c510ea] 2021-03-09T08:48:11.210 INFO:tasks.ceph.mon.a.smithi099.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9766cfeae1] 2021-03-09T08:48:11.210 INFO:tasks.ceph.mon.a.smithi099.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7f9766afe41d] 2021-03-09T08:48:11.210 INFO:tasks.ceph.mon.a.smithi099.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0x7f97644fd14a] 2021-03-09T08:48:11.211 INFO:tasks.ceph.mon.a.smithi099.stderr: 20: clone() 2021-03-09T08:48:11.211 INFO:tasks.ceph.mon.a.smithi099.stderr:2021-03-09T08:48:11.196+0000 7f97590a5700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:48:11.211 INFO:tasks.ceph.mon.a.smithi099.stderr: in thread 7f97590a5700 thread_name:ms_dispatch 2021-03-09T08:48:11.211 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.211 INFO:tasks.ceph.mon.a.smithi099.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:11.212 INFO:tasks.ceph.mon.a.smithi099.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9764507b20] 2021-03-09T08:48:11.212 INFO:tasks.ceph.mon.a.smithi099.stderr: 2: gsignal() 2021-03-09T08:48:11.212 INFO:tasks.ceph.mon.a.smithi099.stderr: 3: abort() 2021-03-09T08:48:11.212 INFO:tasks.ceph.mon.a.smithi099.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f9766a17229] 2021-03-09T08:48:11.213 INFO:tasks.ceph.mon.a.smithi099.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9766a173f2] 2021-03-09T08:48:11.213 INFO:tasks.ceph.mon.a.smithi099.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x562ff8fb697c] 2021-03-09T08:48:11.213 INFO:tasks.ceph.mon.a.smithi099.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x562ff8ff5ced] 2021-03-09T08:48:11.213 INFO:tasks.ceph.mon.a.smithi099.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x562ff901db5a] 2021-03-09T08:48:11.213 INFO:tasks.ceph.mon.a.smithi099.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x562ff9029054] 2021-03-09T08:48:11.214 INFO:tasks.ceph.mon.a.smithi099.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x562ff902d2c3] 2021-03-09T08:48:11.214 INFO:tasks.ceph.mon.a.smithi099.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x562ff8faec2d] 2021-03-09T08:48:11.214 INFO:tasks.ceph.mon.a.smithi099.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x562ff8e97c74] 2021-03-09T08:48:11.215 INFO:tasks.ceph.mon.a.smithi099.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x562ff8e9c4d9] 2021-03-09T08:48:11.215 INFO:tasks.ceph.mon.a.smithi099.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x562ff8e9d766] 2021-03-09T08:48:11.215 INFO:tasks.ceph.mon.a.smithi099.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x562ff8ecbb0c] 2021-03-09T08:48:11.215 INFO:tasks.ceph.mon.a.smithi099.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f9766c510ea] 2021-03-09T08:48:11.215 INFO:tasks.ceph.mon.a.smithi099.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9766cfeae1] 2021-03-09T08:48:11.216 INFO:tasks.ceph.mon.a.smithi099.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7f9766afe41d] 2021-03-09T08:48:11.216 INFO:tasks.ceph.mon.a.smithi099.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0x7f97644fd14a] 2021-03-09T08:48:11.216 INFO:tasks.ceph.mon.a.smithi099.stderr: 20: clone() 2021-03-09T08:48:11.216 INFO:tasks.ceph.mon.a.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:48:11.216 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.217 INFO:tasks.ceph.mon.a.smithi099.stderr: -1> 2021-03-09T08:48:11.194+0000 7f97590a5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f97590a5700 time 2021-03-09T08:48:11.194218+0000 2021-03-09T08:48:11.217 INFO:tasks.ceph.mon.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:48:11.217 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.218 INFO:tasks.ceph.mon.a.smithi099.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:11.218 INFO:tasks.ceph.mon.a.smithi099.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f9766a171d8] 2021-03-09T08:48:11.218 INFO:tasks.ceph.mon.a.smithi099.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9766a173f2] 2021-03-09T08:48:11.218 INFO:tasks.ceph.mon.a.smithi099.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x562ff8fb697c] 2021-03-09T08:48:11.219 INFO:tasks.ceph.mon.a.smithi099.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x562ff8ff5ced] 2021-03-09T08:48:11.219 INFO:tasks.ceph.mon.a.smithi099.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x562ff901db5a] 2021-03-09T08:48:11.219 INFO:tasks.ceph.mon.a.smithi099.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x562ff9029054] 2021-03-09T08:48:11.219 INFO:tasks.ceph.mon.a.smithi099.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x562ff902d2c3] 2021-03-09T08:48:11.219 INFO:tasks.ceph.mon.a.smithi099.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x562ff8faec2d] 2021-03-09T08:48:11.220 INFO:tasks.ceph.mon.a.smithi099.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x562ff8e97c74] 2021-03-09T08:48:11.220 INFO:tasks.ceph.mon.a.smithi099.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x562ff8e9c4d9] 2021-03-09T08:48:11.220 INFO:tasks.ceph.mon.a.smithi099.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x562ff8e9d766] 2021-03-09T08:48:11.221 INFO:tasks.ceph.mon.a.smithi099.stderr: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x562ff8ecbb0c] 2021-03-09T08:48:11.221 INFO:tasks.ceph.mon.a.smithi099.stderr: 13: (DispatchQueue::entry()+0x126a) [0x7f9766c510ea] 2021-03-09T08:48:11.221 INFO:tasks.ceph.mon.a.smithi099.stderr: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9766cfeae1] 2021-03-09T08:48:11.221 INFO:tasks.ceph.mon.a.smithi099.stderr: 15: (Thread::_entry_func(void*)+0xd) [0x7f9766afe41d] 2021-03-09T08:48:11.221 INFO:tasks.ceph.mon.a.smithi099.stderr: 16: /lib64/libpthread.so.0(+0x814a) [0x7f97644fd14a] 2021-03-09T08:48:11.222 INFO:tasks.ceph.mon.a.smithi099.stderr: 17: clone() 2021-03-09T08:48:11.222 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.222 INFO:tasks.ceph.mon.a.smithi099.stderr: 0> 2021-03-09T08:48:11.196+0000 7f97590a5700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:48:11.222 INFO:tasks.ceph.mon.a.smithi099.stderr: in thread 7f97590a5700 thread_name:ms_dispatch 2021-03-09T08:48:11.222 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.223 INFO:tasks.ceph.mon.a.smithi099.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:11.223 INFO:tasks.ceph.mon.a.smithi099.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9764507b20] 2021-03-09T08:48:11.223 INFO:tasks.ceph.mon.a.smithi099.stderr: 2: gsignal() 2021-03-09T08:48:11.223 INFO:tasks.ceph.mon.a.smithi099.stderr: 3: abort() 2021-03-09T08:48:11.224 INFO:tasks.ceph.mon.a.smithi099.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f9766a17229] 2021-03-09T08:48:11.224 INFO:tasks.ceph.mon.a.smithi099.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9766a173f2] 2021-03-09T08:48:11.224 INFO:tasks.ceph.mon.a.smithi099.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x562ff8fb697c] 2021-03-09T08:48:11.224 INFO:tasks.ceph.mon.a.smithi099.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x562ff8ff5ced] 2021-03-09T08:48:11.224 INFO:tasks.ceph.mon.a.smithi099.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x562ff901db5a] 2021-03-09T08:48:11.225 INFO:tasks.ceph.mon.a.smithi099.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x562ff9029054] 2021-03-09T08:48:11.225 INFO:tasks.ceph.mon.a.smithi099.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x562ff902d2c3] 2021-03-09T08:48:11.225 INFO:tasks.ceph.mon.a.smithi099.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x562ff8faec2d] 2021-03-09T08:48:11.225 INFO:tasks.ceph.mon.a.smithi099.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x562ff8e97c74] 2021-03-09T08:48:11.225 INFO:tasks.ceph.mon.a.smithi099.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x562ff8e9c4d9] 2021-03-09T08:48:11.226 INFO:tasks.ceph.mon.a.smithi099.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x562ff8e9d766] 2021-03-09T08:48:11.226 INFO:tasks.ceph.mon.a.smithi099.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x562ff8ecbb0c] 2021-03-09T08:48:11.226 INFO:tasks.ceph.mon.a.smithi099.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f9766c510ea] 2021-03-09T08:48:11.226 INFO:tasks.ceph.mon.a.smithi099.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9766cfeae1] 2021-03-09T08:48:11.226 INFO:tasks.ceph.mon.a.smithi099.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7f9766afe41d] 2021-03-09T08:48:11.227 INFO:tasks.ceph.mon.a.smithi099.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0x7f97644fd14a] 2021-03-09T08:48:11.227 INFO:tasks.ceph.mon.a.smithi099.stderr: 20: clone() 2021-03-09T08:48:11.227 INFO:tasks.ceph.mon.a.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:48:11.227 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.228 INFO:tasks.ceph.mon.a.smithi099.stderr: -1> 2021-03-09T08:48:11.194+0000 7f97590a5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7f97590a5700 time 2021-03-09T08:48:11.194218+0000 2021-03-09T08:48:11.228 INFO:tasks.ceph.mon.a.smithi099.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:48:11.228 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.228 INFO:tasks.ceph.mon.a.smithi099.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:11.229 INFO:tasks.ceph.mon.a.smithi099.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f9766a171d8] 2021-03-09T08:48:11.229 INFO:tasks.ceph.mon.a.smithi099.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9766a173f2] 2021-03-09T08:48:11.229 INFO:tasks.ceph.mon.a.smithi099.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x562ff8fb697c] 2021-03-09T08:48:11.229 INFO:tasks.ceph.mon.a.smithi099.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x562ff8ff5ced] 2021-03-09T08:48:11.230 INFO:tasks.ceph.mon.a.smithi099.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x562ff901db5a] 2021-03-09T08:48:11.230 INFO:tasks.ceph.mon.a.smithi099.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x562ff9029054] 2021-03-09T08:48:11.230 INFO:tasks.ceph.mon.a.smithi099.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x562ff902d2c3] 2021-03-09T08:48:11.230 INFO:tasks.ceph.mon.a.smithi099.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x562ff8faec2d] 2021-03-09T08:48:11.230 INFO:tasks.ceph.mon.a.smithi099.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x562ff8e97c74] 2021-03-09T08:48:11.231 INFO:tasks.ceph.mon.a.smithi099.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x562ff8e9c4d9] 2021-03-09T08:48:11.231 INFO:tasks.ceph.mon.a.smithi099.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x562ff8e9d766] 2021-03-09T08:48:11.231 INFO:tasks.ceph.mon.a.smithi099.stderr: 12: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x562ff8ecbb0c] 2021-03-09T08:48:11.231 INFO:tasks.ceph.mon.a.smithi099.stderr: 13: (DispatchQueue::entry()+0x126a) [0x7f9766c510ea] 2021-03-09T08:48:11.231 INFO:tasks.ceph.mon.a.smithi099.stderr: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9766cfeae1] 2021-03-09T08:48:11.232 INFO:tasks.ceph.mon.a.smithi099.stderr: 15: (Thread::_entry_func(void*)+0xd) [0x7f9766afe41d] 2021-03-09T08:48:11.232 INFO:tasks.ceph.mon.a.smithi099.stderr: 16: /lib64/libpthread.so.0(+0x814a) [0x7f97644fd14a] 2021-03-09T08:48:11.232 INFO:tasks.ceph.mon.a.smithi099.stderr: 17: clone() 2021-03-09T08:48:11.232 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.232 INFO:tasks.ceph.mon.a.smithi099.stderr: 0> 2021-03-09T08:48:11.196+0000 7f97590a5700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:48:11.233 INFO:tasks.ceph.mon.a.smithi099.stderr: in thread 7f97590a5700 thread_name:ms_dispatch 2021-03-09T08:48:11.233 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:11.233 INFO:tasks.ceph.mon.a.smithi099.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:11.233 INFO:tasks.ceph.mon.a.smithi099.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f9764507b20] 2021-03-09T08:48:11.233 INFO:tasks.ceph.mon.a.smithi099.stderr: 2: gsignal() 2021-03-09T08:48:11.234 INFO:tasks.ceph.mon.a.smithi099.stderr: 3: abort() 2021-03-09T08:48:11.234 INFO:tasks.ceph.mon.a.smithi099.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f9766a17229] 2021-03-09T08:48:11.234 INFO:tasks.ceph.mon.a.smithi099.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7f9766a173f2] 2021-03-09T08:48:11.234 INFO:tasks.ceph.mon.a.smithi099.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x562ff8fb697c] 2021-03-09T08:48:11.234 INFO:tasks.ceph.mon.a.smithi099.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x562ff8ff5ced] 2021-03-09T08:48:11.235 INFO:tasks.ceph.mon.a.smithi099.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x562ff901db5a] 2021-03-09T08:48:11.235 INFO:tasks.ceph.mon.a.smithi099.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x562ff9029054] 2021-03-09T08:48:11.235 INFO:tasks.ceph.mon.a.smithi099.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x562ff902d2c3] 2021-03-09T08:48:11.235 INFO:tasks.ceph.mon.a.smithi099.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x562ff8faec2d] 2021-03-09T08:48:11.236 INFO:tasks.ceph.mon.a.smithi099.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x562ff8e97c74] 2021-03-09T08:48:11.236 INFO:tasks.ceph.mon.a.smithi099.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x562ff8e9c4d9] 2021-03-09T08:48:11.236 INFO:tasks.ceph.mon.a.smithi099.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x562ff8e9d766] 2021-03-09T08:48:11.236 INFO:tasks.ceph.mon.a.smithi099.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x562ff8ecbb0c] 2021-03-09T08:48:11.236 INFO:tasks.ceph.mon.a.smithi099.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7f9766c510ea] 2021-03-09T08:48:11.237 INFO:tasks.ceph.mon.a.smithi099.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9766cfeae1] 2021-03-09T08:48:11.237 INFO:tasks.ceph.mon.a.smithi099.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7f9766afe41d] 2021-03-09T08:48:11.237 INFO:tasks.ceph.mon.a.smithi099.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0x7f97644fd14a] 2021-03-09T08:48:11.237 INFO:tasks.ceph.mon.a.smithi099.stderr: 20: clone() 2021-03-09T08:48:11.238 INFO:tasks.ceph.mon.a.smithi099.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:48:11.238 INFO:tasks.ceph.mon.a.smithi099.stderr: 2021-03-09T08:48:12.415 INFO:tasks.ceph.mon.a.smithi099.stderr:daemon-helper: command crashed with signal 6 2021-03-09T08:48:12.446 DEBUG:teuthology.orchestra.run.smithi099:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-03-09T08:48:12.578 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.576+0000 7f71b91d2700 10 monclient: get_monmap_and_config 2021-03-09T08:48:12.579 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.576+0000 7f71b91d2700 10 monclient: build_initial_monmap 2021-03-09T08:48:12.579 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.576+0000 7f71b91d2700 10 monclient: monmap: 2021-03-09T08:48:12.579 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2021-03-09T08:48:12.580 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:12.580 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:48:12.577630+0000 2021-03-09T08:48:12.580 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:48:12.577630+0000 2021-03-09T08:48:12.580 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:12.581 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:12.581 INFO:teuthology.misc.health.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:12.581 INFO:teuthology.misc.health.smithi099.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:12.581 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:12.581 INFO:teuthology.misc.health.smithi099.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:12.582 INFO:teuthology.misc.health.smithi099.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:12.582 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:12.594 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 1 Processor -- start 2021-03-09T08:48:12.594 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 1 -- start start 2021-03-09T08:48:12.594 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient: init 2021-03-09T08:48:12.594 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:12.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient: _add_conns ranks=[3,0,4,1,2] 2021-03-09T08:48:12.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f71b40a9900 0x7f71b40a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:12.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient(hunting): picked mon.noname-a con 0x7f71b40a9900 addr v2:172.21.15.99:3300/0 2021-03-09T08:48:12.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71b4139be0 addr v1:172.21.15.99:6789/0 2021-03-09T08:48:12.595 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f71b40a41d0 0x7f71b40a4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:12.596 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient(hunting): picked mon.noname-c con 0x7f71b40a41d0 addr v2:172.21.15.195:3300/0 2021-03-09T08:48:12.596 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40a90f0 con 0x7f71b4139be0 2021-03-09T08:48:12.596 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:12.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b40a80d0 con 0x7f71b40a9900 2021-03-09T08:48:12.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:12.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b40a9390 con 0x7f71b40a41d0 2021-03-09T08:48:12.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:12.597 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b91d2700 10 monclient(hunting): authenticate will time out at 859.052846s 2021-03-09T08:48:12.598 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b3fff700 1 -- >> v2:172.21.15.99:3300/0 conn(0x7f71b40a9900 msgr2=0x7f71b40a7be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:48:12.598 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b37fe700 1 -- >> v1:172.21.15.99:6789/0 conn(0x7f71b4139be0 legacy=0x7f71b413a340 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2021-03-09T08:48:12.598 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.577+0000 7f71b3fff700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f71b40a9900 0x7f71b40a7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:12.598 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f71b40a41d0 0x7f71b40a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:12.599 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f71b40a41d0 0x7f71b40a4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.195:3300/0 says I am v2:172.21.15.99:58220/0 (socket says 172.21.15.99:58220) 2021-03-09T08:48:12.599 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 1 -- 172.21.15.99:0/3871276029 learned_addr learned my addr 172.21.15.99:0/3871276029 (peer_addr_for_me v2:172.21.15.99:0/0) 2021-03-09T08:48:12.599 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 10 monclient(hunting): get_auth_request con 0x7f71b40a41d0 auth_method 0 2021-03-09T08:48:12.599 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:12.600 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:12.600 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:12.600 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:12.600 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:12.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 10 monclient(hunting): handle_auth_done global_id 4249 payload 450 2021-03-09T08:48:12.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 1 -- 172.21.15.99:0/3871276029 >> v2:172.21.15.99:3300/0 conn(0x7f71b40a9900 msgr2=0x7f71b40a7be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:12.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 1 --2- 172.21.15.99:0/3871276029 >> v2:172.21.15.99:3300/0 conn(0x7f71b40a9900 0x7f71b40a7be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:12.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 1 -- 172.21.15.99:0/3871276029 >> v1:172.21.15.99:6789/0 conn(0x7f71b4139be0 legacy=0x7f71b413a340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:12.601 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 10 monclient: _finish_hunting 0 2021-03-09T08:48:12.602 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 1 monclient: found mon.noname-c 2021-03-09T08:48:12.602 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:12.602 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.195:3300/0 2021-03-09T08:48:12.602 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 1 -- 172.21.15.99:0/3871276029 --> v2:172.21.15.195:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b40a82f0 con 0x7f71b40a41d0 2021-03-09T08:48:12.602 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b89d1700 1 --2- 172.21.15.99:0/3871276029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b40a41d0 0x7f71b40a4660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f71b40aa2a0 tx=0x7f71a80092d0).ready entity=mon.1 client_cookie=f75b598b3f3d917 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:12.603 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b2ffd700 1 -- 172.21.15.99:0/3871276029 <== mon.1 v2:172.21.15.195:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a800bbb0 con 0x7f71b40a41d0 2021-03-09T08:48:12.603 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:12.603 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T08:48:12.603 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b2ffd700 10 monclient: dump: 2021-03-09T08:48:12.603 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2021-03-09T08:48:12.604 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:12.604 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:12.604 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:12.604 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:12.604 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:12.605 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:12.605 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:12.605 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:12.605 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:12.605 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b2ffd700 10 monclient: _finish_auth 0 2021-03-09T08:48:12.606 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:12.606 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b91d2700 5 monclient: authenticate success, global_id 4249 2021-03-09T08:48:12.606 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b2ffd700 1 -- 172.21.15.99:0/3871276029 <== mon.1 v2:172.21.15.195:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a800bd10 con 0x7f71b40a41d0 2021-03-09T08:48:12.606 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.578+0000 7f71b91d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:12.606 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b2ffd700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:12.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:12.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 4 set_mon_vals no callback set 2021-03-09T08:48:12.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b2ffd700 1 -- 172.21.15.99:0/3871276029 <== mon.1 v2:172.21.15.195:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a8011610 con 0x7f71b40a41d0 2021-03-09T08:48:12.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:12.607 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2021-03-09T08:48:12.608 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b2ffd700 10 monclient: dump: 2021-03-09T08:48:12.608 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2021-03-09T08:48:12.608 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:12.608 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:12.608 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:12.609 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:12.609 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:12.609 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:12.610 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:12.610 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:12.610 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:12.611 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 10 monclient: shutdown 2021-03-09T08:48:12.611 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:12.611 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 1 -- 172.21.15.99:0/3871276029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b40a41d0 msgr2=0x7f71b40a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:12.611 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 1 --2- 172.21.15.99:0/3871276029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b40a41d0 0x7f71b40a4660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f71b40aa2a0 tx=0x7f71a80092d0).stop 2021-03-09T08:48:12.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 1 -- 172.21.15.99:0/3871276029 shutdown_connections 2021-03-09T08:48:12.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 1 --2- 172.21.15.99:0/3871276029 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b40a41d0 0x7f71b40a4660 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:12.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 1 --2- 172.21.15.99:0/3871276029 >> v2:172.21.15.99:3300/0 conn(0x7f71b40a9900 0x7f71b40a7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:12.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 1 -- 172.21.15.99:0/3871276029 >> 172.21.15.99:0/3871276029 conn(0x7f71b4064350 msgr2=0x7f71b4069f90 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:12.612 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 1 -- 172.21.15.99:0/3871276029 shutdown_connections 2021-03-09T08:48:12.613 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.579+0000 7f71b91d2700 1 -- 172.21.15.99:0/3871276029 wait complete. 2021-03-09T08:48:12.613 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.580+0000 7f71b91d2700 10 monclient: build_initial_monmap 2021-03-09T08:48:12.613 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.580+0000 7f71b91d2700 10 monclient: monmap: 2021-03-09T08:48:12.613 INFO:teuthology.misc.health.smithi099.stderr:epoch 0 2021-03-09T08:48:12.613 INFO:teuthology.misc.health.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:12.614 INFO:teuthology.misc.health.smithi099.stderr:last_changed 0.000000 2021-03-09T08:48:12.614 INFO:teuthology.misc.health.smithi099.stderr:created 0.000000 2021-03-09T08:48:12.614 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:12.614 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 1 2021-03-09T08:48:12.614 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:12.615 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:12.615 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:12.615 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:12.615 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.580+0000 7f71b91d2700 1 librados: starting msgr at 2021-03-09T08:48:12.616 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.580+0000 7f71b91d2700 1 librados: starting objecter 2021-03-09T08:48:12.616 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.580+0000 7f71b91d2700 1 Processor -- start 2021-03-09T08:48:12.616 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.580+0000 7f71b91d2700 1 -- start start 2021-03-09T08:48:12.616 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.580+0000 7f71b91d2700 1 librados: setting wanted keys 2021-03-09T08:48:12.617 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.580+0000 7f71b91d2700 1 librados: calling monclient init 2021-03-09T08:48:12.617 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.580+0000 7f71b91d2700 10 monclient: init 2021-03-09T08:48:12.617 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:12.617 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:48:12.618 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f71b40a41d0 0x7f71b4146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:12.618 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient(hunting): picked mon.noname-c con 0x7f71b40a41d0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:12.618 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71b40a9900 0x7f71b41455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:12.618 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient(hunting): picked mon.noname-a con 0x7f71b40a9900 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:12.619 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b40687e0 0x7f71b4114b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:12.619 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient(hunting): picked mon.noname-b con 0x7f71b40687e0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:12.619 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:12.619 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4145c80 con 0x7f71b40a9900 2021-03-09T08:48:12.619 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:12.620 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4132aa0 con 0x7f71b40687e0 2021-03-09T08:48:12.620 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:12.620 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4115010 con 0x7f71b40a41d0 2021-03-09T08:48:12.620 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:12.621 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient(hunting): authenticate will time out at 859.056175s 2021-03-09T08:48:12.621 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b37fe700 1 -- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71b40a9900 msgr2=0x7f71b41455e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:48:12.621 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b37fe700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71b40a9900 0x7f71b41455e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:12.621 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f71b40a41d0 0x7f71b4146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:12.621 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f71b40a41d0 0x7f71b4146120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.99:3301/0 says I am v2:172.21.15.99:56872/0 (socket says 172.21.15.99:56872) 2021-03-09T08:48:12.622 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 -- 172.21.15.99:0/281728241 learned_addr learned my addr 172.21.15.99:0/281728241 (peer_addr_for_me v2:172.21.15.99:0/0) 2021-03-09T08:48:12.622 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 10 monclient(hunting): get_auth_request con 0x7f71b40a41d0 auth_method 0 2021-03-09T08:48:12.622 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:12.622 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:12.623 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:12.623 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:12.623 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b89d1700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b40687e0 0x7f71b4114b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:12.623 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:12.623 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b89d1700 10 monclient(hunting): get_auth_request con 0x7f71b40687e0 auth_method 0 2021-03-09T08:48:12.624 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b89d1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:12.624 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b89d1700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:12.624 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b89d1700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:12.624 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b89d1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:12.624 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b89d1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:12.625 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 10 monclient(hunting): handle_auth_done global_id 4202 payload 995 2021-03-09T08:48:12.625 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 -- 172.21.15.99:0/281728241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b40687e0 msgr2=0x7f71b4114b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:12.625 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b40687e0 0x7f71b4114b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:12.625 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 -- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71b40a9900 msgr2=0x7f71b41455e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:48:12.625 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71b40a9900 0x7f71b41455e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:12.626 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 10 monclient: _finish_hunting 0 2021-03-09T08:48:12.626 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 monclient: found mon.noname-c 2021-03-09T08:48:12.626 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:12.626 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.99:3301/0 2021-03-09T08:48:12.626 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 -- 172.21.15.99:0/281728241 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b4148b40 con 0x7f71b40a41d0 2021-03-09T08:48:12.627 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b3fff700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f71b40a41d0 0x7f71b4146120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f71a0009710 tx=0x7f71a0006e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:12.627 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 client.?.objecter ms_handle_connect 0x7f71b40a41d0 2021-03-09T08:48:12.627 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:12.627 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 1 -- 172.21.15.99:0/281728241 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a0006170 con 0x7f71b40a41d0 2021-03-09T08:48:12.627 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:12.628 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T08:48:12.628 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 monclient: dump: 2021-03-09T08:48:12.628 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2021-03-09T08:48:12.628 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:12.628 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:12.629 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:12.629 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:12.629 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:12.629 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:12.630 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:12.630 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:12.630 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:12.630 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 monclient: _finish_auth 0 2021-03-09T08:48:12.630 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:12.631 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 5 monclient: authenticate success, global_id 4202 2021-03-09T08:48:12.631 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 1 -- 172.21.15.99:0/281728241 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a00062d0 con 0x7f71b40a41d0 2021-03-09T08:48:12.631 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient: _renew_subs 2021-03-09T08:48:12.631 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:12.631 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b4148df0 con 0x7f71b40a41d0 2021-03-09T08:48:12.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:12.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 1 -- 172.21.15.99:0/281728241 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71a0003910 con 0x7f71b40a41d0 2021-03-09T08:48:12.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:12.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:48:12.632 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 client.4202.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:12.633 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f719ffff700 10 monclient: dump: 2021-03-09T08:48:12.633 INFO:teuthology.misc.health.smithi099.stderr:epoch 1 2021-03-09T08:48:12.633 INFO:teuthology.misc.health.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:12.633 INFO:teuthology.misc.health.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:12.634 INFO:teuthology.misc.health.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:12.634 INFO:teuthology.misc.health.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:12.634 INFO:teuthology.misc.health.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:12.634 INFO:teuthology.misc.health.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:12.634 INFO:teuthology.misc.health.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:12.635 INFO:teuthology.misc.health.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:12.635 INFO:teuthology.misc.health.smithi099.stderr: 2021-03-09T08:48:12.635 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b27fc700 4 set_mon_vals no callback set 2021-03-09T08:48:12.635 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient: _renew_subs 2021-03-09T08:48:12.635 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:12.636 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b4149300 con 0x7f71b40a41d0 2021-03-09T08:48:12.636 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.581+0000 7f71b91d2700 1 librados: init done 2021-03-09T08:48:12.636 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.582+0000 7f71b91d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:12.636 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.582+0000 7f71b91d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:12.636 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.582+0000 7f71b91d2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:12.637 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.582+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b4145ad0 con 0x7f71b40a41d0 2021-03-09T08:48:12.637 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f719ffff700 1 -- 172.21.15.99:0/281728241 <== mon.2 v2:172.21.15.99:3301/0 4 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f71a001a060 con 0x7f71b40a41d0 2021-03-09T08:48:12.637 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f719ffff700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f719405ba00 0x7f719405dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:12.637 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f719ffff700 1 -- 172.21.15.99:0/281728241 <== mon.2 v2:172.21.15.99:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f71a0072f40 con 0x7f71b40a41d0 2021-03-09T08:48:12.637 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f719ffff700 10 client.4202.objecter ms_dispatch 0x7f71b4001550 osd_map(13..13 src has 1..13) v4 2021-03-09T08:48:12.638 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f719ffff700 3 client.4202.objecter handle_osd_map got epochs [13,13] > 0 2021-03-09T08:48:12.638 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f719ffff700 3 client.4202.objecter handle_osd_map decoding full epoch 13 2021-03-09T08:48:12.638 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f71b37fe700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f719405ba00 0x7f719405dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:12.638 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f71b37fe700 10 monclient: get_auth_request con 0x7f719405ba00 auth_method 0 2021-03-09T08:48:12.638 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f719ffff700 20 client.4202.objecter dump_active .. 0 homeless 2021-03-09T08:48:12.639 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f719ffff700 10 client.4202.objecter ms_handle_connect 0x7f719405ba00 2021-03-09T08:48:12.639 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.583+0000 7f71b37fe700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f719405ba00 0x7f719405dec0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f71b4068200 tx=0x7f71a4006ac0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:12.639 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.587+0000 7f719ffff700 1 -- 172.21.15.99:0/281728241 <== mon.2 v2:172.21.15.99:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+149001 (secure 0 0 0) 0x7f71a004a930 con 0x7f71b40a41d0 2021-03-09T08:48:12.639 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.587+0000 7f719ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:12.639 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.587+0000 7f719ffff700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:12.738 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.736+0000 7f71b91d2700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2021-03-09T08:48:12.738 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.736+0000 7f71b91d2700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2021-03-09T08:48:12.738 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.736+0000 7f71b91d2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:12.739 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:12.736+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71b405d170 con 0x7f71b40a41d0 2021-03-09T08:48:15.583 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:15.581+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:15.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:15.581+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:15.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:15.581+0000 7f719effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:16.583 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:16.581+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:16.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:16.581+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:17.583 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:17.582+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:17.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:17.582+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:17.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:17.582+0000 7f71b0ff9700 10 client.4202.objecter tick 2021-03-09T08:48:18.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:18.582+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:18.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:18.582+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:19.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:19.581+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:19.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:19.581+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:20.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:20.582+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:20.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:20.582+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:21.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:21.582+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:21.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:21.582+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:22.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:22.582+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:22.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:22.582+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:22.585 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:22.583+0000 7f71b0ff9700 10 client.4202.objecter tick 2021-03-09T08:48:23.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:23.583+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:23.585 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:23.583+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:24.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:24.582+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:24.585 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:24.582+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:25.584 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:25.583+0000 7f719effd700 10 monclient: tick 2021-03-09T08:48:25.585 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:25.583+0000 7f719effd700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:25.585 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:25.583+0000 7f719effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:26.175 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.172+0000 7f719ffff700 1 -- 172.21.15.99:0/281728241 <== mon.2 v2:172.21.15.99:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10406 (secure 0 0 0) 0x7f71a0018070 con 0x7f71b40a41d0 2021-03-09T08:48:26.175 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.173+0000 7f719ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2021-03-09T08:48:26.176 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.173+0000 7f719ffff700 10 monclient: _finish_command 2 = system:0 2021-03-09T08:48:26.176 INFO:teuthology.misc.health.smithi099.stdout: 2021-03-09T08:48:26.176 INFO:teuthology.misc.health.smithi099.stdout:{"epoch":13,"fsid":"e7417d55-b73a-46ac-8f0e-ad6e583f612d","created":"2021-03-09T08:48:02.065911+0000","modified":"2021-03-09T08:48:11.156832+0000","last_up_change":"2021-03-09T08:48:11.156832+0000","last_in_change":"2021-03-09T08:48:04.540876+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"bfcce70f-6a3c-4d76-9a45-145c7d5e468d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":30122},{"type":"v1","addr":"172.21.15.99:6801","nonce":30122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":30122},{"type":"v1","addr":"172.21.15.99:6803","nonce":30122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":30122},{"type":"v1","addr":"172.21.15.99:6807","nonce":30122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":30122},{"type":"v1","addr":"172.21.15.99:6805","nonce":30122}]},"public_addr":"172.21.15.99:6801/30122","cluster_addr":"172.21.15.99:6803/30122","heartbeat_back_addr":"172.21.15.99:6807/30122","heartbeat_front_addr":"172.21.15.99:6805/30122","state":["exists","up"]},{"osd":1,"uuid":"ff4574a3-9580-4767-b99d-f2d4e488b465","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":30124},{"type":"v1","addr":"172.21.15.99:6809","nonce":30124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6811","nonce":30124},{"type":"v1","addr":"172.21.15.99:6813","nonce":30124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":30124},{"type":"v1","addr":"172.21.15.99:6823","nonce":30124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6815","nonce":30124},{"type":"v1","addr":"172.21.15.99:6817","nonce":30124}]},"public_addr":"172.21.15.99:6809/30124","cluster_addr":"172.21.15.99:6813/30124","heartbeat_back_addr":"172.21.15.99:6823/30124","heartbeat_front_addr":"172.21.15.99:6817/30124","state":["exists","up"]},{"osd":2,"uuid":"eb8b7997-db54-48ba-8336-f8566c57204f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":30123},{"type":"v1","addr":"172.21.15.99:6812","nonce":30123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":30123},{"type":"v1","addr":"172.21.15.99:6816","nonce":30123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":30123},{"type":"v1","addr":"172.21.15.99:6826","nonce":30123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":30123},{"type":"v1","addr":"172.21.15.99:6821","nonce":30123}]},"public_addr":"172.21.15.99:6812/30123","cluster_addr":"172.21.15.99:6816/30123","heartbeat_back_addr":"172.21.15.99:6826/30123","heartbeat_front_addr":"172.21.15.99:6821/30123","state":["exists","up"]},{"osd":3,"uuid":"dc1a81b3-b2c9-4c37-9429-6d131c9e791c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6819","nonce":30139},{"type":"v1","addr":"172.21.15.99:6822","nonce":30139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6825","nonce":30139},{"type":"v1","addr":"172.21.15.99:6827","nonce":30139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":30139},{"type":"v1","addr":"172.21.15.99:6831","nonce":30139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":30139},{"type":"v1","addr":"172.21.15.99:6829","nonce":30139}]},"public_addr":"172.21.15.99:6822/30139","cluster_addr":"172.21.15.99:6827/30139","heartbeat_back_addr":"172.21.15.99:6831/30139","heartbeat_front_addr":"172.21.15.99:6829/30139","state":["exists","up"]},{"osd":4,"uuid":"802be6f1-4e15-4e2d-8720-0a3009f8d02e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":30064},{"type":"v1","addr":"172.21.15.195:6801","nonce":30064}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":30064},{"type":"v1","addr":"172.21.15.195:6803","nonce":30064}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":30064},{"type":"v1","addr":"172.21.15.195:6810","nonce":30064}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":30064},{"type":"v1","addr":"172.21.15.195:6806","nonce":30064}]},"public_addr":"172.21.15.195:6801/30064","cluster_addr":"172.21.15.195:6803/30064","heartbeat_back_addr":"172.21.15.195:6810/30064","heartbeat_front_addr":"172.21.15.195:6806/30064","state":["exists","up"]},{"osd":5,"uuid":"17c2bea4-c1fc-49d2-a191-957e9575fd9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6805","nonce":30065},{"type":"v1","addr":"172.21.15.195:6807","nonce":30065}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6809","nonce":30065},{"type":"v1","addr":"172.21.15.195:6811","nonce":30065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":30065},{"type":"v1","addr":"172.21.15.195:6815","nonce":30065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":30065},{"type":"v1","addr":"172.21.15.195:6813","nonce":30065}]},"public_addr":"172.21.15.195:6807/30065","cluster_addr":"172.21.15.195:6811/30065","heartbeat_back_addr":"172.21.15.195:6815/30065","heartbeat_front_addr":"172.21.15.195:6813/30065","state":["exists","up"]},{"osd":6,"uuid":"27bb6c24-657c-4887-b344-aa7c2dd5676f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":30067},{"type":"v1","addr":"172.21.15.195:6825","nonce":30067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":30067},{"type":"v1","addr":"172.21.15.195:6827","nonce":30067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":30067},{"type":"v1","addr":"172.21.15.195:6831","nonce":30067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":30067},{"type":"v1","addr":"172.21.15.195:6829","nonce":30067}]},"public_addr":"172.21.15.195:6825/30067","cluster_addr":"172.21.15.195:6827/30067","heartbeat_back_addr":"172.21.15.195:6831/30067","heartbeat_front_addr":"172.21.15.195:6829/30067","state":["exists","up"]},{"osd":7,"uuid":"b11e00a8-aaeb-49b0-89a4-9720edf16c13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":30066},{"type":"v1","addr":"172.21.15.195:6817","nonce":30066}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":30066},{"type":"v1","addr":"172.21.15.195:6819","nonce":30066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":30066},{"type":"v1","addr":"172.21.15.195:6823","nonce":30066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":30066},{"type":"v1","addr":"172.21.15.195:6821","nonce":30066}]},"public_addr":"172.21.15.195:6817/30066","cluster_addr":"172.21.15.195:6819/30066","heartbeat_back_addr":"172.21.15.195:6823/30066","heartbeat_front_addr":"172.21.15.195:6821/30066","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-09T08:48:26.178 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 10 librados: watch_flush enter 2021-03-09T08:48:26.178 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 10 librados: watch_flush exit 2021-03-09T08:48:26.178 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 20 client.4202.objecter shutdown clearing up homeless session... 2021-03-09T08:48:26.179 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 10 client.4202.objecter successfully canceled tick 2021-03-09T08:48:26.179 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f719405ba00 msgr2=0x7f719405dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:26.179 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f719405ba00 0x7f719405dec0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f71b4068200 tx=0x7f71a4006ac0).stop 2021-03-09T08:48:26.179 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 10 monclient: shutdown 2021-03-09T08:48:26.180 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:26.180 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f71b40a41d0 msgr2=0x7f71b4146120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:26.180 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f71b40a41d0 0x7f71b4146120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f71a0009710 tx=0x7f71a0006e50).stop 2021-03-09T08:48:26.180 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 shutdown_connections 2021-03-09T08:48:26.180 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f719405ba00 0x7f719405dec0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:26.181 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f71b40687e0 0x7f71b4114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:26.181 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f71b40a41d0 0x7f71b4146120 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:26.181 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 --2- 172.21.15.99:0/281728241 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f71b40a9900 0x7f71b41455e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:26.181 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 >> 172.21.15.99:0/281728241 conn(0x7f71b4064350 msgr2=0x7f71b4139fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:26.181 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 shutdown_connections 2021-03-09T08:48:26.182 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 -- 172.21.15.99:0/281728241 wait complete. 2021-03-09T08:48:26.182 INFO:teuthology.misc.health.smithi099.stderr:2021-03-09T08:48:26.175+0000 7f71b91d2700 1 librados: shutdown 2021-03-09T08:48:26.191 DEBUG:teuthology.misc:8 of 8 OSDs are up 2021-03-09T08:48:26.196 INFO:tasks.ceph:Creating RBD pool 2021-03-09T08:48:26.197 DEBUG:teuthology.orchestra.run.smithi099:> sudo ceph --cluster ceph osd pool create rbd 8 2021-03-09T08:48:26.199 INFO:tasks.ceph.mon.b.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7ffbbd829700 time 2021-03-09T08:48:26.198127+0000 2021-03-09T08:48:26.199 INFO:tasks.ceph.mon.b.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:48:26.200 INFO:tasks.ceph.mon.b.smithi195.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:26.201 INFO:tasks.ceph.mon.b.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ffbcb19b1d8] 2021-03-09T08:48:26.201 INFO:tasks.ceph.mon.b.smithi195.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7ffbcb19b3f2] 2021-03-09T08:48:26.202 INFO:tasks.ceph.mon.b.smithi195.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x561a0325397c] 2021-03-09T08:48:26.202 INFO:tasks.ceph.mon.b.smithi195.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x561a03292ced] 2021-03-09T08:48:26.203 INFO:tasks.ceph.mon.b.smithi195.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x561a032bab5a] 2021-03-09T08:48:26.203 INFO:tasks.ceph.mon.b.smithi195.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x561a032c6054] 2021-03-09T08:48:26.204 INFO:tasks.ceph.mon.b.smithi195.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x561a032ca2c3] 2021-03-09T08:48:26.204 INFO:tasks.ceph.mon.b.smithi195.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x561a0324bc2d] 2021-03-09T08:48:26.204 INFO:tasks.ceph.mon.b.smithi195.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x561a03134c74] 2021-03-09T08:48:26.205 INFO:tasks.ceph.mon.b.smithi195.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x561a031394d9] 2021-03-09T08:48:26.205 INFO:tasks.ceph.mon.b.smithi195.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.205 INFO:tasks.ceph.mon.b.smithi195.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x561a0313bf91] 2021-03-09T08:48:26.205 INFO:tasks.ceph.mon.b.smithi195.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x561a03139f46] 2021-03-09T08:48:26.206 INFO:tasks.ceph.mon.b.smithi195.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.206 INFO:tasks.ceph.mon.b.smithi195.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x561a03168b0c] 2021-03-09T08:48:26.207 INFO:tasks.ceph.mon.b.smithi195.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7ffbcb3d50ea] 2021-03-09T08:48:26.207 INFO:tasks.ceph.mon.b.smithi195.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ffbcb482ae1] 2021-03-09T08:48:26.207 INFO:tasks.ceph.mon.b.smithi195.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7ffbcb28241d] 2021-03-09T08:48:26.207 INFO:tasks.ceph.mon.b.smithi195.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0x7ffbc8c8114a] 2021-03-09T08:48:26.207 INFO:tasks.ceph.mon.b.smithi195.stderr: 20: clone() 2021-03-09T08:48:26.208 INFO:tasks.ceph.mon.b.smithi195.stderr:*** Caught signal (Aborted) ** 2021-03-09T08:48:26.208 INFO:tasks.ceph.mon.b.smithi195.stderr: in thread 7ffbbd829700 thread_name:ms_dispatch 2021-03-09T08:48:26.208 INFO:tasks.ceph.mon.b.smithi195.stderr:2021-03-09T08:48:26.199+0000 7ffbbd829700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7ffbbd829700 time 2021-03-09T08:48:26.198127+0000 2021-03-09T08:48:26.208 INFO:tasks.ceph.mon.b.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:48:26.209 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.209 INFO:tasks.ceph.mon.b.smithi195.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:26.209 INFO:tasks.ceph.mon.b.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ffbcb19b1d8] 2021-03-09T08:48:26.209 INFO:tasks.ceph.mon.b.smithi195.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7ffbcb19b3f2] 2021-03-09T08:48:26.210 INFO:tasks.ceph.mon.b.smithi195.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x561a0325397c] 2021-03-09T08:48:26.210 INFO:tasks.ceph.mon.b.smithi195.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x561a03292ced] 2021-03-09T08:48:26.210 INFO:tasks.ceph.mon.b.smithi195.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x561a032bab5a] 2021-03-09T08:48:26.210 INFO:tasks.ceph.mon.b.smithi195.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x561a032c6054] 2021-03-09T08:48:26.211 INFO:tasks.ceph.mon.b.smithi195.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x561a032ca2c3] 2021-03-09T08:48:26.211 INFO:tasks.ceph.mon.b.smithi195.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x561a0324bc2d] 2021-03-09T08:48:26.211 INFO:tasks.ceph.mon.b.smithi195.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x561a03134c74] 2021-03-09T08:48:26.211 INFO:tasks.ceph.mon.b.smithi195.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x561a031394d9] 2021-03-09T08:48:26.212 INFO:tasks.ceph.mon.b.smithi195.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.212 INFO:tasks.ceph.mon.b.smithi195.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x561a0313bf91] 2021-03-09T08:48:26.212 INFO:tasks.ceph.mon.b.smithi195.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x561a03139f46] 2021-03-09T08:48:26.212 INFO:tasks.ceph.mon.b.smithi195.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.213 INFO:tasks.ceph.mon.b.smithi195.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x561a03168b0c] 2021-03-09T08:48:26.213 INFO:tasks.ceph.mon.b.smithi195.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7ffbcb3d50ea] 2021-03-09T08:48:26.213 INFO:tasks.ceph.mon.b.smithi195.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ffbcb482ae1] 2021-03-09T08:48:26.213 INFO:tasks.ceph.mon.b.smithi195.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7ffbcb28241d] 2021-03-09T08:48:26.214 INFO:tasks.ceph.mon.b.smithi195.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0x7ffbc8c8114a] 2021-03-09T08:48:26.214 INFO:tasks.ceph.mon.b.smithi195.stderr: 20: clone() 2021-03-09T08:48:26.214 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.215 INFO:tasks.ceph.mon.b.smithi195.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:26.215 INFO:tasks.ceph.mon.b.smithi195.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ffbc8c8bb20] 2021-03-09T08:48:26.215 INFO:tasks.ceph.mon.b.smithi195.stderr: 2: gsignal() 2021-03-09T08:48:26.215 INFO:tasks.ceph.mon.b.smithi195.stderr: 3: abort() 2021-03-09T08:48:26.216 INFO:tasks.ceph.mon.b.smithi195.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7ffbcb19b229] 2021-03-09T08:48:26.216 INFO:tasks.ceph.mon.b.smithi195.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7ffbcb19b3f2] 2021-03-09T08:48:26.216 INFO:tasks.ceph.mon.b.smithi195.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x561a0325397c] 2021-03-09T08:48:26.216 INFO:tasks.ceph.mon.b.smithi195.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x561a03292ced] 2021-03-09T08:48:26.216 INFO:tasks.ceph.mon.b.smithi195.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x561a032bab5a] 2021-03-09T08:48:26.217 INFO:tasks.ceph.mon.b.smithi195.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x561a032c6054] 2021-03-09T08:48:26.218 INFO:tasks.ceph.mon.b.smithi195.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x561a032ca2c3] 2021-03-09T08:48:26.218 INFO:tasks.ceph.mon.b.smithi195.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x561a0324bc2d] 2021-03-09T08:48:26.218 INFO:tasks.ceph.mon.b.smithi195.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x561a03134c74] 2021-03-09T08:48:26.218 INFO:tasks.ceph.mon.b.smithi195.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x561a031394d9] 2021-03-09T08:48:26.218 INFO:tasks.ceph.mon.b.smithi195.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.219 INFO:tasks.ceph.mon.b.smithi195.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x561a0313bf91] 2021-03-09T08:48:26.219 INFO:tasks.ceph.mon.b.smithi195.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x561a03139f46] 2021-03-09T08:48:26.219 INFO:tasks.ceph.mon.b.smithi195.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.219 INFO:tasks.ceph.mon.b.smithi195.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x561a03168b0c] 2021-03-09T08:48:26.220 INFO:tasks.ceph.mon.b.smithi195.stderr: 19: (DispatchQueue::entry()+0x126a) [0x7ffbcb3d50ea] 2021-03-09T08:48:26.220 INFO:tasks.ceph.mon.b.smithi195.stderr: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ffbcb482ae1] 2021-03-09T08:48:26.220 INFO:tasks.ceph.mon.b.smithi195.stderr: 21: (Thread::_entry_func(void*)+0xd) [0x7ffbcb28241d] 2021-03-09T08:48:26.220 INFO:tasks.ceph.mon.b.smithi195.stderr: 22: /lib64/libpthread.so.0(+0x814a) [0x7ffbc8c8114a] 2021-03-09T08:48:26.221 INFO:tasks.ceph.mon.b.smithi195.stderr: 23: clone() 2021-03-09T08:48:26.221 INFO:tasks.ceph.mon.b.smithi195.stderr:2021-03-09T08:48:26.201+0000 7ffbbd829700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:48:26.221 INFO:tasks.ceph.mon.b.smithi195.stderr: in thread 7ffbbd829700 thread_name:ms_dispatch 2021-03-09T08:48:26.221 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.222 INFO:tasks.ceph.mon.b.smithi195.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:26.222 INFO:tasks.ceph.mon.b.smithi195.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ffbc8c8bb20] 2021-03-09T08:48:26.222 INFO:tasks.ceph.mon.b.smithi195.stderr: 2: gsignal() 2021-03-09T08:48:26.222 INFO:tasks.ceph.mon.b.smithi195.stderr: 3: abort() 2021-03-09T08:48:26.223 INFO:tasks.ceph.mon.b.smithi195.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7ffbcb19b229] 2021-03-09T08:48:26.223 INFO:tasks.ceph.mon.b.smithi195.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7ffbcb19b3f2] 2021-03-09T08:48:26.223 INFO:tasks.ceph.mon.b.smithi195.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x561a0325397c] 2021-03-09T08:48:26.223 INFO:tasks.ceph.mon.b.smithi195.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x561a03292ced] 2021-03-09T08:48:26.224 INFO:tasks.ceph.mon.b.smithi195.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x561a032bab5a] 2021-03-09T08:48:26.224 INFO:tasks.ceph.mon.b.smithi195.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x561a032c6054] 2021-03-09T08:48:26.224 INFO:tasks.ceph.mon.b.smithi195.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x561a032ca2c3] 2021-03-09T08:48:26.224 INFO:tasks.ceph.mon.b.smithi195.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x561a0324bc2d] 2021-03-09T08:48:26.225 INFO:tasks.ceph.mon.b.smithi195.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x561a03134c74] 2021-03-09T08:48:26.225 INFO:tasks.ceph.mon.b.smithi195.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x561a031394d9] 2021-03-09T08:48:26.225 INFO:tasks.ceph.mon.b.smithi195.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.225 INFO:tasks.ceph.mon.b.smithi195.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x561a0313bf91] 2021-03-09T08:48:26.226 INFO:tasks.ceph.mon.b.smithi195.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x561a03139f46] 2021-03-09T08:48:26.226 INFO:tasks.ceph.mon.b.smithi195.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.226 INFO:tasks.ceph.mon.b.smithi195.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x561a03168b0c] 2021-03-09T08:48:26.227 INFO:tasks.ceph.mon.b.smithi195.stderr: 19: (DispatchQueue::entry()+0x126a) [0x7ffbcb3d50ea] 2021-03-09T08:48:26.227 INFO:tasks.ceph.mon.b.smithi195.stderr: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ffbcb482ae1] 2021-03-09T08:48:26.227 INFO:tasks.ceph.mon.b.smithi195.stderr: 21: (Thread::_entry_func(void*)+0xd) [0x7ffbcb28241d] 2021-03-09T08:48:26.227 INFO:tasks.ceph.mon.b.smithi195.stderr: 22: /lib64/libpthread.so.0(+0x814a) [0x7ffbc8c8114a] 2021-03-09T08:48:26.228 INFO:tasks.ceph.mon.b.smithi195.stderr: 23: clone() 2021-03-09T08:48:26.228 INFO:tasks.ceph.mon.b.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:48:26.228 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.229 INFO:tasks.ceph.mon.b.smithi195.stderr: -1> 2021-03-09T08:48:26.199+0000 7ffbbd829700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7ffbbd829700 time 2021-03-09T08:48:26.198127+0000 2021-03-09T08:48:26.230 INFO:tasks.ceph.mon.b.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:48:26.230 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.230 INFO:tasks.ceph.mon.b.smithi195.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:26.231 INFO:tasks.ceph.mon.b.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ffbcb19b1d8] 2021-03-09T08:48:26.231 INFO:tasks.ceph.mon.b.smithi195.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7ffbcb19b3f2] 2021-03-09T08:48:26.231 INFO:tasks.ceph.mon.b.smithi195.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x561a0325397c] 2021-03-09T08:48:26.231 INFO:tasks.ceph.mon.b.smithi195.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x561a03292ced] 2021-03-09T08:48:26.231 INFO:tasks.ceph.mon.b.smithi195.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x561a032bab5a] 2021-03-09T08:48:26.232 INFO:tasks.ceph.mon.b.smithi195.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x561a032c6054] 2021-03-09T08:48:26.232 INFO:tasks.ceph.mon.b.smithi195.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x561a032ca2c3] 2021-03-09T08:48:26.232 INFO:tasks.ceph.mon.b.smithi195.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x561a0324bc2d] 2021-03-09T08:48:26.232 INFO:tasks.ceph.mon.b.smithi195.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x561a03134c74] 2021-03-09T08:48:26.233 INFO:tasks.ceph.mon.b.smithi195.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x561a031394d9] 2021-03-09T08:48:26.233 INFO:tasks.ceph.mon.b.smithi195.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.233 INFO:tasks.ceph.mon.b.smithi195.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x561a0313bf91] 2021-03-09T08:48:26.233 INFO:tasks.ceph.mon.b.smithi195.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x561a03139f46] 2021-03-09T08:48:26.234 INFO:tasks.ceph.mon.b.smithi195.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.234 INFO:tasks.ceph.mon.b.smithi195.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x561a03168b0c] 2021-03-09T08:48:26.234 INFO:tasks.ceph.mon.b.smithi195.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7ffbcb3d50ea] 2021-03-09T08:48:26.234 INFO:tasks.ceph.mon.b.smithi195.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ffbcb482ae1] 2021-03-09T08:48:26.235 INFO:tasks.ceph.mon.b.smithi195.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7ffbcb28241d] 2021-03-09T08:48:26.235 INFO:tasks.ceph.mon.b.smithi195.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0x7ffbc8c8114a] 2021-03-09T08:48:26.235 INFO:tasks.ceph.mon.b.smithi195.stderr: 20: clone() 2021-03-09T08:48:26.235 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.235 INFO:tasks.ceph.mon.b.smithi195.stderr: 0> 2021-03-09T08:48:26.201+0000 7ffbbd829700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:48:26.236 INFO:tasks.ceph.mon.b.smithi195.stderr: in thread 7ffbbd829700 thread_name:ms_dispatch 2021-03-09T08:48:26.236 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.236 INFO:tasks.ceph.mon.b.smithi195.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:26.236 INFO:tasks.ceph.mon.b.smithi195.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ffbc8c8bb20] 2021-03-09T08:48:26.236 INFO:tasks.ceph.mon.b.smithi195.stderr: 2: gsignal() 2021-03-09T08:48:26.237 INFO:tasks.ceph.mon.b.smithi195.stderr: 3: abort() 2021-03-09T08:48:26.237 INFO:tasks.ceph.mon.b.smithi195.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7ffbcb19b229] 2021-03-09T08:48:26.237 INFO:tasks.ceph.mon.b.smithi195.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7ffbcb19b3f2] 2021-03-09T08:48:26.238 INFO:tasks.ceph.mon.b.smithi195.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x561a0325397c] 2021-03-09T08:48:26.238 INFO:tasks.ceph.mon.b.smithi195.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x561a03292ced] 2021-03-09T08:48:26.238 INFO:tasks.ceph.mon.b.smithi195.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x561a032bab5a] 2021-03-09T08:48:26.238 INFO:tasks.ceph.mon.b.smithi195.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x561a032c6054] 2021-03-09T08:48:26.238 INFO:tasks.ceph.mon.b.smithi195.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x561a032ca2c3] 2021-03-09T08:48:26.239 INFO:tasks.ceph.mon.b.smithi195.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x561a0324bc2d] 2021-03-09T08:48:26.239 INFO:tasks.ceph.mon.b.smithi195.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x561a03134c74] 2021-03-09T08:48:26.239 INFO:tasks.ceph.mon.b.smithi195.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x561a031394d9] 2021-03-09T08:48:26.239 INFO:tasks.ceph.mon.b.smithi195.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.240 INFO:tasks.ceph.mon.b.smithi195.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x561a0313bf91] 2021-03-09T08:48:26.240 INFO:tasks.ceph.mon.b.smithi195.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x561a03139f46] 2021-03-09T08:48:26.240 INFO:tasks.ceph.mon.b.smithi195.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.240 INFO:tasks.ceph.mon.b.smithi195.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x561a03168b0c] 2021-03-09T08:48:26.240 INFO:tasks.ceph.mon.b.smithi195.stderr: 19: (DispatchQueue::entry()+0x126a) [0x7ffbcb3d50ea] 2021-03-09T08:48:26.241 INFO:tasks.ceph.mon.b.smithi195.stderr: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ffbcb482ae1] 2021-03-09T08:48:26.241 INFO:tasks.ceph.mon.b.smithi195.stderr: 21: (Thread::_entry_func(void*)+0xd) [0x7ffbcb28241d] 2021-03-09T08:48:26.241 INFO:tasks.ceph.mon.b.smithi195.stderr: 22: /lib64/libpthread.so.0(+0x814a) [0x7ffbc8c8114a] 2021-03-09T08:48:26.241 INFO:tasks.ceph.mon.b.smithi195.stderr: 23: clone() 2021-03-09T08:48:26.241 INFO:tasks.ceph.mon.b.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:48:26.242 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.242 INFO:tasks.ceph.mon.b.smithi195.stderr: -1> 2021-03-09T08:48:26.199+0000 7ffbbd829700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: In function 'mempool::osdmap_mapping::vector& OSDMapMapping::get_osd_acting_pgs(unsigned int)' thread 7ffbbd829700 time 2021-03-09T08:48:26.198127+0000 2021-03-09T08:48:26.242 INFO:tasks.ceph.mon.b.smithi195.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1742-g89fb622b/rpm/el8/BUILD/ceph-17.0.0-1742-g89fb622b/src/osd/OSDMapMapping.h: 325: FAILED ceph_assert(osd < acting_rmap.size()) 2021-03-09T08:48:26.242 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.242 INFO:tasks.ceph.mon.b.smithi195.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:26.243 INFO:tasks.ceph.mon.b.smithi195.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7ffbcb19b1d8] 2021-03-09T08:48:26.243 INFO:tasks.ceph.mon.b.smithi195.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7ffbcb19b3f2] 2021-03-09T08:48:26.243 INFO:tasks.ceph.mon.b.smithi195.stderr: 3: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x561a0325397c] 2021-03-09T08:48:26.243 INFO:tasks.ceph.mon.b.smithi195.stderr: 4: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x561a03292ced] 2021-03-09T08:48:26.243 INFO:tasks.ceph.mon.b.smithi195.stderr: 5: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x561a032bab5a] 2021-03-09T08:48:26.244 INFO:tasks.ceph.mon.b.smithi195.stderr: 6: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x561a032c6054] 2021-03-09T08:48:26.244 INFO:tasks.ceph.mon.b.smithi195.stderr: 7: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x561a032ca2c3] 2021-03-09T08:48:26.244 INFO:tasks.ceph.mon.b.smithi195.stderr: 8: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x561a0324bc2d] 2021-03-09T08:48:26.244 INFO:tasks.ceph.mon.b.smithi195.stderr: 9: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x561a03134c74] 2021-03-09T08:48:26.245 INFO:tasks.ceph.mon.b.smithi195.stderr: 10: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x561a031394d9] 2021-03-09T08:48:26.245 INFO:tasks.ceph.mon.b.smithi195.stderr: 11: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.245 INFO:tasks.ceph.mon.b.smithi195.stderr: 12: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x561a0313bf91] 2021-03-09T08:48:26.245 INFO:tasks.ceph.mon.b.smithi195.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x561a03139f46] 2021-03-09T08:48:26.245 INFO:tasks.ceph.mon.b.smithi195.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.246 INFO:tasks.ceph.mon.b.smithi195.stderr: 15: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x561a03168b0c] 2021-03-09T08:48:26.246 INFO:tasks.ceph.mon.b.smithi195.stderr: 16: (DispatchQueue::entry()+0x126a) [0x7ffbcb3d50ea] 2021-03-09T08:48:26.246 INFO:tasks.ceph.mon.b.smithi195.stderr: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ffbcb482ae1] 2021-03-09T08:48:26.246 INFO:tasks.ceph.mon.b.smithi195.stderr: 18: (Thread::_entry_func(void*)+0xd) [0x7ffbcb28241d] 2021-03-09T08:48:26.246 INFO:tasks.ceph.mon.b.smithi195.stderr: 19: /lib64/libpthread.so.0(+0x814a) [0x7ffbc8c8114a] 2021-03-09T08:48:26.247 INFO:tasks.ceph.mon.b.smithi195.stderr: 20: clone() 2021-03-09T08:48:26.247 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.247 INFO:tasks.ceph.mon.b.smithi195.stderr: 0> 2021-03-09T08:48:26.201+0000 7ffbbd829700 -1 *** Caught signal (Aborted) ** 2021-03-09T08:48:26.247 INFO:tasks.ceph.mon.b.smithi195.stderr: in thread 7ffbbd829700 thread_name:ms_dispatch 2021-03-09T08:48:26.247 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.248 INFO:tasks.ceph.mon.b.smithi195.stderr: ceph version 17.0.0-1742-g89fb622b (89fb622b10ca2e58fe0c913a8956377a886e2ab4) quincy (dev) 2021-03-09T08:48:26.248 INFO:tasks.ceph.mon.b.smithi195.stderr: 1: /lib64/libpthread.so.0(+0x12b20) [0x7ffbc8c8bb20] 2021-03-09T08:48:26.248 INFO:tasks.ceph.mon.b.smithi195.stderr: 2: gsignal() 2021-03-09T08:48:26.248 INFO:tasks.ceph.mon.b.smithi195.stderr: 3: abort() 2021-03-09T08:48:26.248 INFO:tasks.ceph.mon.b.smithi195.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7ffbcb19b229] 2021-03-09T08:48:26.249 INFO:tasks.ceph.mon.b.smithi195.stderr: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2763f2) [0x7ffbcb19b3f2] 2021-03-09T08:48:26.249 INFO:tasks.ceph.mon.b.smithi195.stderr: 6: (OSDMonitor::check_pg_num(long, int, int, int, std::ostream*)+0x4fc) [0x561a0325397c] 2021-03-09T08:48:26.249 INFO:tasks.ceph.mon.b.smithi195.stderr: 7: (OSDMonitor::prepare_new_pool(std::__cxx11::basic_string, std::allocator >&, int, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned int, unsigned int, unsigned long, unsigned long, float, std::__cxx11::basic_string, std::allocator > const&, unsigned int, unsigned long, OSDMonitor::FastReadType, std::__cxx11::basic_string, std::allocator > const&, std::ostream*)+0x6cd) [0x561a03292ced] 2021-03-09T08:48:26.249 INFO:tasks.ceph.mon.b.smithi195.stderr: 8: (OSDMonitor::prepare_command_impl(boost::intrusive_ptr, std::map, std::allocator >, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > >, std::less, std::allocator, std::allocator > const, boost::variant, std::allocator >, bool, long, double, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector >, std::vector > > > > > const&)+0x1845a) [0x561a032bab5a] 2021-03-09T08:48:26.249 INFO:tasks.ceph.mon.b.smithi195.stderr: 9: (OSDMonitor::prepare_command(boost::intrusive_ptr)+0xf4) [0x561a032c6054] 2021-03-09T08:48:26.250 INFO:tasks.ceph.mon.b.smithi195.stderr: 10: (OSDMonitor::prepare_update(boost::intrusive_ptr)+0x373) [0x561a032ca2c3] 2021-03-09T08:48:26.250 INFO:tasks.ceph.mon.b.smithi195.stderr: 11: (PaxosService::dispatch(boost::intrusive_ptr)+0xa6d) [0x561a0324bc2d] 2021-03-09T08:48:26.250 INFO:tasks.ceph.mon.b.smithi195.stderr: 12: (Monitor::handle_command(boost::intrusive_ptr)+0x2794) [0x561a03134c74] 2021-03-09T08:48:26.250 INFO:tasks.ceph.mon.b.smithi195.stderr: 13: (Monitor::dispatch_op(boost::intrusive_ptr)+0x7f9) [0x561a031394d9] 2021-03-09T08:48:26.250 INFO:tasks.ceph.mon.b.smithi195.stderr: 14: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.251 INFO:tasks.ceph.mon.b.smithi195.stderr: 15: (Monitor::handle_forward(boost::intrusive_ptr)+0x941) [0x561a0313bf91] 2021-03-09T08:48:26.251 INFO:tasks.ceph.mon.b.smithi195.stderr: 16: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1266) [0x561a03139f46] 2021-03-09T08:48:26.251 INFO:tasks.ceph.mon.b.smithi195.stderr: 17: (Monitor::_ms_dispatch(Message*)+0x5f6) [0x561a0313a766] 2021-03-09T08:48:26.251 INFO:tasks.ceph.mon.b.smithi195.stderr: 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x561a03168b0c] 2021-03-09T08:48:26.251 INFO:tasks.ceph.mon.b.smithi195.stderr: 19: (DispatchQueue::entry()+0x126a) [0x7ffbcb3d50ea] 2021-03-09T08:48:26.252 INFO:tasks.ceph.mon.b.smithi195.stderr: 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ffbcb482ae1] 2021-03-09T08:48:26.252 INFO:tasks.ceph.mon.b.smithi195.stderr: 21: (Thread::_entry_func(void*)+0xd) [0x7ffbcb28241d] 2021-03-09T08:48:26.252 INFO:tasks.ceph.mon.b.smithi195.stderr: 22: /lib64/libpthread.so.0(+0x814a) [0x7ffbc8c8114a] 2021-03-09T08:48:26.252 INFO:tasks.ceph.mon.b.smithi195.stderr: 23: clone() 2021-03-09T08:48:26.252 INFO:tasks.ceph.mon.b.smithi195.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-09T08:48:26.253 INFO:tasks.ceph.mon.b.smithi195.stderr: 2021-03-09T08:48:26.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.331+0000 7f6407fb6700 10 monclient: get_monmap_and_config 2021-03-09T08:48:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.331+0000 7f6407fb6700 10 monclient: build_initial_monmap 2021-03-09T08:48:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.331+0000 7f6407fb6700 10 monclient: monmap: 2021-03-09T08:48:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 0 2021-03-09T08:48:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:48:26.332869+0000 2021-03-09T08:48:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:48:26.332869+0000 2021-03-09T08:48:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:0: v1:172.21.15.99:6789/0 mon.noname-a-legacy 2021-03-09T08:48:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:1: v1:172.21.15.195:6789/0 mon.noname-c-legacy 2021-03-09T08:48:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-b 2021-03-09T08:48:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:3: v2:172.21.15.99:3300/0 mon.noname-a 2021-03-09T08:48:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:4: v2:172.21.15.195:3300/0 mon.noname-c 2021-03-09T08:48:26.337 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:26.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 1 Processor -- start 2021-03-09T08:48:26.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 1 -- start start 2021-03-09T08:48:26.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 10 monclient: init 2021-03-09T08:48:26.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:26.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2021-03-09T08:48:26.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64000ada30 addr v1:172.21.15.99:6789/0 2021-03-09T08:48:26.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 0x7f64000a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:26.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 10 monclient(hunting): picked mon.noname-a con 0x7f64000a6040 addr v2:172.21.15.99:3300/0 2021-03-09T08:48:26.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 0x7f6400139410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:26.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.332+0000 7f6407fb6700 10 monclient(hunting): picked mon.noname-c con 0x7f6400138f80 addr v2:172.21.15.195:3300/0 2021-03-09T08:48:26.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6407fb6700 1 -- --> v1:172.21.15.99:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64000ade10 con 0x7f64000ada30 2021-03-09T08:48:26.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6407fb6700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:26.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6407fb6700 1 -- --> v2:172.21.15.99:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64000a8dd0 con 0x7f64000a6040 2021-03-09T08:48:26.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6407fb6700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:26.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6407fb6700 1 -- --> v2:172.21.15.195:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64000a8ac0 con 0x7f6400138f80 2021-03-09T08:48:26.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6407fb6700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:26.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6407fb6700 10 monclient(hunting): authenticate will time out at 872.808024s 2021-03-09T08:48:26.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6405d52700 1 -- >> v1:172.21.15.99:6789/0 conn(0x7f64000ada30 legacy=0x7f64000a5930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2021-03-09T08:48:26.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6405551700 1 -- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 msgr2=0x7f64000a6be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:48:26.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6405551700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 0x7f64000a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:26.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6406553700 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 msgr2=0x7f6400139410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:48:26.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.333+0000 7f6406553700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 0x7f6400139410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:26.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.534+0000 7f6405551700 1 -- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 msgr2=0x7f64000a6be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:48:26.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.534+0000 7f6405d52700 1 -- >> v1:172.21.15.99:6789/0 conn(0x7f64000ada30 legacy=0x7f64000a5930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2021-03-09T08:48:26.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.534+0000 7f6405551700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 0x7f64000a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:48:26.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.534+0000 7f6406553700 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 msgr2=0x7f6400139410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:48:26.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.534+0000 7f6406553700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 0x7f6400139410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T08:48:26.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.935+0000 7f6405551700 1 -- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 msgr2=0x7f64000a6be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:48:26.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.935+0000 7f6405d52700 1 -- >> v1:172.21.15.99:6789/0 conn(0x7f64000ada30 legacy=0x7f64000a5930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2021-03-09T08:48:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.935+0000 7f6405551700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 0x7f64000a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:48:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.935+0000 7f6406553700 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 msgr2=0x7f6400139410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:48:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:26.935+0000 7f6406553700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 0x7f6400139410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2021-03-09T08:48:27.471 INFO:tasks.ceph.mon.b.smithi195.stderr:daemon-helper: command crashed with signal 6 2021-03-09T08:48:27.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:27.736+0000 7f6405551700 1 -- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 msgr2=0x7f64000a6be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:48:27.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:27.736+0000 7f6405d52700 1 -- >> v1:172.21.15.99:6789/0 conn(0x7f64000ada30 legacy=0x7f64000a5930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.99:6789/0 2021-03-09T08:48:27.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:27.736+0000 7f6405551700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 0x7f64000a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:48:27.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:27.736+0000 7f6406553700 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 msgr2=0x7f6400139410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:48:27.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:27.736+0000 7f6406553700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 0x7f6400139410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2021-03-09T08:48:29.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 10 monclient(hunting): tick 2021-03-09T08:48:29.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 1 monclient(hunting): continuing hunt 2021-03-09T08:48:29.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T08:48:29.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 1 -- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 msgr2=0x7f6400139410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:29.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 1 --2- >> v2:172.21.15.195:3300/0 conn(0x7f6400138f80 0x7f6400139410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:29.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 1 -- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 msgr2=0x7f64000a6be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:29.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 1 --2- >> v2:172.21.15.99:3300/0 conn(0x7f64000a6040 0x7f64000a6be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:29.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 1 -- >> v1:172.21.15.99:6789/0 conn(0x7f64000ada30 legacy=0x7f64000a5930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 10 monclient: _add_conns ranks=[1,2] 2021-03-09T08:48:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63e00033b0 addr v1:172.21.15.195:6789/0 2021-03-09T08:48:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e00070a0 0x7f63e0009560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 10 monclient(hunting): picked mon.noname-b con 0x7f63e00070a0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 1 -- --> v1:172.21.15.195:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64000a4480 con 0x7f63e00033b0 2021-03-09T08:48:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64000a8dd0 con 0x7f63e00070a0 2021-03-09T08:48:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f63ef7fe700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405d52700 1 -- >> v1:172.21.15.195:6789/0 conn(0x7f63e00033b0 legacy=0x7f63e0005830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.195:6789/0 2021-03-09T08:48:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e00070a0 0x7f63e0009560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e00070a0 0x7f63e0009560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.99:3301/0 says I am v2:172.21.15.99:56928/0 (socket says 172.21.15.99:56928) 2021-03-09T08:48:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 1 -- 172.21.15.99:0/3783775818 learned_addr learned my addr 172.21.15.99:0/3783775818 (peer_addr_for_me v2:172.21.15.99:0/0) 2021-03-09T08:48:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e00070a0 auth_method 0 2021-03-09T08:48:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4205 payload 450 2021-03-09T08:48:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 1 -- 172.21.15.99:0/3783775818 >> v1:172.21.15.195:6789/0 conn(0x7f63e00033b0 legacy=0x7f63e0005830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2021-03-09T08:48:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:48:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 1 monclient: found mon.noname-b 2021-03-09T08:48:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.99:3301/0 2021-03-09T08:48:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.333+0000 7f6405551700 1 -- 172.21.15.99:0/3783775818 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e000ac50 con 0x7f63e00070a0 2021-03-09T08:48:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6405551700 1 --2- 172.21.15.99:0/3783775818 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e00070a0 0x7f63e0009560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f63f4009be0 tx=0x7f63f40091d0).ready entity=mon.2 client_cookie=68cde0c5a589bd56 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 1 -- 172.21.15.99:0/3783775818 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003b70 con 0x7f63e00070a0 2021-03-09T08:48:29.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:29.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2021-03-09T08:48:29.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 10 monclient: dump: 2021-03-09T08:48:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:29.357 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:29.357 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:29.357 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:29.357 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:29.358 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 10 monclient: _finish_auth 0 2021-03-09T08:48:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 1 -- 172.21.15.99:0/3783775818 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003cd0 con 0x7f63e00070a0 2021-03-09T08:48:29.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6407fb6700 5 monclient: authenticate success, global_id 4205 2021-03-09T08:48:29.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6407fb6700 20 monclient: get_monmap_and_config waiting for monmap|config 2021-03-09T08:48:29.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:29.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6407fb6700 10 monclient: get_monmap_and_config success 2021-03-09T08:48:29.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 1 -- 172.21.15.99:0/3783775818 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005320 con 0x7f63e00070a0 2021-03-09T08:48:29.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:29.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6407fb6700 4 set_mon_vals no callback set 2021-03-09T08:48:29.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:48:29.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.334+0000 7f6404d50700 10 monclient: dump: 2021-03-09T08:48:29.361 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:29.362 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:29.362 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:29.362 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:29.362 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:29.363 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:29.363 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:29.363 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:29.363 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:29.364 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:29.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.335+0000 7f6407fb6700 10 monclient: shutdown 2021-03-09T08:48:29.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.335+0000 7f6407fb6700 20 monclient: shutdown discarding 0 pending message(s) 2021-03-09T08:48:29.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.335+0000 7f6407fb6700 1 -- 172.21.15.99:0/3783775818 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e00070a0 msgr2=0x7f63e0009560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T08:48:29.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.335+0000 7f6407fb6700 1 --2- 172.21.15.99:0/3783775818 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e00070a0 0x7f63e0009560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f63f4009be0 tx=0x7f63f40091d0).stop 2021-03-09T08:48:29.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.335+0000 7f6406553700 1 -- 172.21.15.99:0/3783775818 reap_dead start 2021-03-09T08:48:29.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.335+0000 7f6407fb6700 1 -- 172.21.15.99:0/3783775818 shutdown_connections 2021-03-09T08:48:29.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.335+0000 7f6407fb6700 1 -- 172.21.15.99:0/3783775818 >> 172.21.15.99:0/3783775818 conn(0x7f64000631d0 msgr2=0x7f64000635b0 unknown :-1 s=STATE_NONE l=0).mark_down 2021-03-09T08:48:29.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.335+0000 7f6407fb6700 1 -- 172.21.15.99:0/3783775818 shutdown_connections 2021-03-09T08:48:29.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.335+0000 7f6407fb6700 1 -- 172.21.15.99:0/3783775818 wait complete. 2021-03-09T08:48:29.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.336+0000 7f6407fb6700 10 monclient: build_initial_monmap 2021-03-09T08:48:29.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.336+0000 7f6407fb6700 10 monclient: monmap: 2021-03-09T08:48:29.367 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 0 2021-03-09T08:48:29.367 INFO:teuthology.orchestra.run.smithi099.stderr:fsid 00000000-0000-0000-0000-000000000000 2021-03-09T08:48:29.367 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 0.000000 2021-03-09T08:48:29.367 INFO:teuthology.orchestra.run.smithi099.stderr:created 0.000000 2021-03-09T08:48:29.368 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 0 (unknown) 2021-03-09T08:48:29.368 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 1 2021-03-09T08:48:29.368 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.noname-a 2021-03-09T08:48:29.368 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.noname-b 2021-03-09T08:48:29.369 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.noname-c 2021-03-09T08:48:29.369 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:29.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.336+0000 7f6407fb6700 1 librados: starting msgr at 2021-03-09T08:48:29.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.336+0000 7f6407fb6700 1 librados: starting objecter 2021-03-09T08:48:29.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.336+0000 7f6407fb6700 1 Processor -- start 2021-03-09T08:48:29.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.336+0000 7f6407fb6700 1 -- start start 2021-03-09T08:48:29.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.336+0000 7f6407fb6700 1 librados: setting wanted keys 2021-03-09T08:48:29.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.336+0000 7f6407fb6700 1 librados: calling monclient init 2021-03-09T08:48:29.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.336+0000 7f6407fb6700 10 monclient: init 2021-03-09T08:48:29.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:29.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:48:29.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f6400115d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:29.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient(hunting): picked mon.noname-a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:29.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f6400136d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:29.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient(hunting): picked mon.noname-b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:29.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f64000a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:29.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient(hunting): picked mon.noname-c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:29.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:29.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 1 -- --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64000a19b0 con 0x7f63e00070a0 2021-03-09T08:48:29.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:29.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 1 -- --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64001469f0 con 0x7f6400145bd0 2021-03-09T08:48:29.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:29.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 1 -- --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6400116230 con 0x7f6400146050 2021-03-09T08:48:29.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:29.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6405d52700 1 -- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f6400115d40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:48:29.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6407fb6700 10 monclient(hunting): authenticate will time out at 875.812257s 2021-03-09T08:48:29.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6405d52700 1 --2- >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f6400115d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:29.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6405551700 1 -- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f6400136d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:48:29.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f64000a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:29.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6405551700 1 --2- >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f6400136d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:29.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 --2- >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f64000a23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.99:3301/0 says I am v2:172.21.15.99:56934/0 (socket says 172.21.15.99:56934) 2021-03-09T08:48:29.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 learned_addr learned my addr 172.21.15.99:0/2209380211 (peer_addr_for_me v2:172.21.15.99:0/0) 2021-03-09T08:48:29.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:48:29.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:29.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:29.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:29.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:29.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:29.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:48:29.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f6400136d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:48:29.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f6400136d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:29.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f6400115d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:48:29.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f6400115d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:29.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:48:29.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 monclient: found mon.noname-c 2021-03-09T08:48:29.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:29.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.99:3301/0 2021-03-09T08:48:29.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6400137210 con 0x7f6400146050 2021-03-09T08:48:29.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f64000a23b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f63e0007070 tx=0x7f63fc0090f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:29.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f63edffb700 10 client.?.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:48:29.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.337+0000 7f63edffb700 10 client.?.objecter resend_mon_ops 2021-03-09T08:48:29.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc003d10 con 0x7f6400146050 2021-03-09T08:48:29.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:29.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2021-03-09T08:48:29.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:48:29.381 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:29.381 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:29.381 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:29.381 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:29.382 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:29.382 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:29.382 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:29.382 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:29.382 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:29.383 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:29.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 10 monclient: _finish_auth 0 2021-03-09T08:48:29.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:29.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc011030 con 0x7f6400146050 2021-03-09T08:48:29.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:29.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc006320 con 0x7f6400146050 2021-03-09T08:48:29.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f6407fb6700 5 monclient: authenticate success, global_id 4208 2021-03-09T08:48:29.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f6407fb6700 10 monclient: _renew_subs 2021-03-09T08:48:29.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f6407fb6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:29.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:48:29.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f6407fb6700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64001374c0 con 0x7f6400146050 2021-03-09T08:48:29.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:29.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:48:29.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:48:29.386 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:29.386 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:29.386 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:29.386 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:29.387 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:29.387 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:29.387 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:29.387 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:29.387 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:29.388 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:29.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f6407fb6700 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2021-03-09T08:48:29.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f6407fb6700 10 monclient: _renew_subs 2021-03-09T08:48:29.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f6407fb6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:29.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f6407fb6700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64000a2a60 con 0x7f6400146050 2021-03-09T08:48:29.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.338+0000 7f6407fb6700 1 librados: init done 2021-03-09T08:48:29.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.339+0000 7f6407fb6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:29.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.339+0000 7f6407fb6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:29.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.339+0000 7f6407fb6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:29.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.339+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 4 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0066d0 con 0x7f6400146050 2021-03-09T08:48:29.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.339+0000 7f6407fb6700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640005e040 con 0x7f6400146050 2021-03-09T08:48:29.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.339+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e005c7c0 0x7f63e005ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:29.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f63fc072b20 con 0x7f6400146050 2021-03-09T08:48:29.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.340+0000 7f63edffb700 10 client.4208.objecter ms_dispatch 0x7f6400001550 osd_map(13..13 src has 1..13) v4 2021-03-09T08:48:29.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.340+0000 7f63edffb700 3 client.4208.objecter handle_osd_map got epochs [13,13] > 0 2021-03-09T08:48:29.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.340+0000 7f63edffb700 3 client.4208.objecter handle_osd_map decoding full epoch 13 2021-03-09T08:48:29.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e005c7c0 0x7f63e005ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:29.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.340+0000 7f6405d52700 10 monclient: get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T08:48:29.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.340+0000 7f63edffb700 20 client.4208.objecter dump_active .. 0 homeless 2021-03-09T08:48:29.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.340+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T08:48:29.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e005c7c0 0x7f63e005ec80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f63f00036d0 tx=0x7f63f0006b30).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:29.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+149001 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T08:48:29.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.344+0000 7f63edffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2021-03-09T08:48:29.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.344+0000 7f63edffb700 10 monclient: _finish_command 1 = system:0 2021-03-09T08:48:29.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.499+0000 7f6407fb6700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:48:29.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.499+0000 7f6407fb6700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:48:29.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.499+0000 7f6407fb6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:29.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:29.499+0000 7f6407fb6700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f64001163c0 con 0x7f6400146050 2021-03-09T08:48:32.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:32.338+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:32.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:32.338+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:32.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:32.338+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:33.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:33.338+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:33.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:33.338+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:34.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:34.338+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:34.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:34.338+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:34.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:34.339+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:48:35.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:35.338+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:35.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:35.338+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:36.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:36.338+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:36.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:36.338+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:37.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:37.338+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:37.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:37.338+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:38.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:38.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:38.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:38.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:39.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:39.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:39.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:39.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:39.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:39.340+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:48:40.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:40.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:40.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:40.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:41.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:41.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:41.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:41.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:42.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:42.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:42.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:42.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:42.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:42.339+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:43.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:43.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:43.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:43.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:44.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:44.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:44.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:44.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:44.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:44.341+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:48:45.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:45.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:45.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:45.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:45.840 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:48:45.839+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20 slow ops, oldest is mon_command({"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1} v 0) 2021-03-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:46.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:46.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:46.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:47.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:47.340+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:47.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:47.340+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:48.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:48.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:48.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:49.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:49.340+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:49.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:49.340+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:49.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:49.342+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:48:50.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.340+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:50.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.340+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:50.841 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:48:50.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f64000a23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:48:50.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f64000a23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:48:50.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f64000a23b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f63e0007070 tx=0x7f63fc0090f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:48:50.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f64000a23b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f63e0007070 tx=0x7f63fc0090f0).stop 2021-03-09T08:48:50.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:50.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:50.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f64000a23b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:48:50.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f64000a23b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:50.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:48:50.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:50.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:50.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0066ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:50.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:50.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e00692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:50.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:50.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:50.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069860 con 0x7f63e00070a0 2021-03-09T08:48:50.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:50.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00699a0 con 0x7f6400145bd0 2021-03-09T08:48:50.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:50.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069b30 con 0x7f63e0064730 2021-03-09T08:48:50.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.839+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:50.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.840+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e00692c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:48:50.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.840+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e00692c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:50.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.840+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:48:50.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.840+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:50.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.841+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0066ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.841+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:48:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.841+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.841+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.841+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.841+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.841+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:48:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:48:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e00692c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:48:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e00692c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:48:50.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:48:50.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:50.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:50.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0069cf0 con 0x7f63e0064730 2021-03-09T08:48:50.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:48:50.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:50.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f640005e040 con 0x7f63e0064730 2021-03-09T08:48:50.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:48:50.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:50.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0066ba0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f64000a4b60 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:50.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:48:50.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.842+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:48:50.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc077020 con 0x7f63e0064730 2021-03-09T08:48:50.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.844+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.844+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:48:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.844+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:48:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:50.858 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01dd00 con 0x7f63e0064730 2021-03-09T08:48:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.845+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.845+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:48:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:50.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049c90 con 0x7f63e0064730 2021-03-09T08:48:51.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:51.339+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:51.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:51.339+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:52.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:52.340+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:52.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:52.340+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:52.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:52.340+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:53.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:53.341+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:53.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:53.341+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:54.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:54.341+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:54.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:54.341+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:54.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:54.343+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:48:55.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.341+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:55.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.341+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:55.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0066ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:48:55.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0066ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:48:55.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0066ba0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f64000a4b60 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:48:55.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0066ba0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f64000a4b60 tx=0x7f63fc04a740).stop 2021-03-09T08:48:55.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:55.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:48:55.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0066ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:48:55.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0066ba0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:48:55.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:48:55.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:55.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:48:55.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e00692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:55.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:48:55.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:48:55.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:55.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e00692c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:48:55.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e00692c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:48:55.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:48:55.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:48:55.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:55.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002630 con 0x7f63e00070a0 2021-03-09T08:48:55.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:55.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006ee0 con 0x7f6400145bd0 2021-03-09T08:48:55.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:48:55.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d790 con 0x7f6400146050 2021-03-09T08:48:55.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:48:55.849 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:48:55.840+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:48:55.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:48:55.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.841+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:48:55.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.841+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:48:55.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.841+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:48:55.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.842+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:48:55.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.842+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:48:55.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.842+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:48:55.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:48:55.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:48:55.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:55.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e00692c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:48:55.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e00692c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:48:55.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:48:55.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:48:55.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:48:55.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:55.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006c420 con 0x7f6400146050 2021-03-09T08:48:55.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:48:55.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:48:55.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63f00159e0 con 0x7f6400146050 2021-03-09T08:48:55.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:48:55.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.844+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:55.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.846+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bf30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f63f0015d00 tx=0x7f63f0013380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:48:55.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.846+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:48:55.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.846+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:48:55.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013da0 con 0x7f6400146050 2021-03-09T08:48:55.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.848+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:48:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.848+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:48:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.848+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:48:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:48:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:48:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:48:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:48:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:48:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:48:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:48:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:48:55.859 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:48:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001a070 con 0x7f6400146050 2021-03-09T08:48:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.848+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:48:55.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.848+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:48:55.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:55.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f000f070 con 0x7f6400146050 2021-03-09T08:48:56.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:56.341+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:56.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:56.341+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:57.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:57.341+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:57.341+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:58.341+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:58.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:58.341+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:59.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:59.341+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:48:59.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:59.341+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:48:59.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:48:59.344+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:00.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.341+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:00.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.341+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:00.842 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:49:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bf30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f63f0015d00 tx=0x7f63f0013380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bf30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f63f0015d00 tx=0x7f63f0013380).stop 2021-03-09T08:49:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006bf30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bf30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:00.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:49:00.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:00.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:00.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:00.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:00.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:00.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:00.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:00.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006da0 con 0x7f63e00070a0 2021-03-09T08:49:00.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:00.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002160 con 0x7f6400145bd0 2021-03-09T08:49:00.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:00.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000ef0 con 0x7f63e0064730 2021-03-09T08:49:00.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.841+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:00.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:00.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.842+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:00.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:00.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:00.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.842+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:00.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.842+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:49:00.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.842+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:00.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.842+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:00.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.843+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:00.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.843+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:00.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.843+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:00.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:00.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:00.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:00.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:00.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:00.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:49:00.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:49:00.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:00.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:00.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066f00 con 0x7f63e0064730 2021-03-09T08:49:00.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:00.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:00.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f640005e040 con 0x7f63e0064730 2021-03-09T08:49:00.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:49:00.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.844+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:00.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.845+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002b580 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f63fc010e20 tx=0x7f63fc00baa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:00.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.845+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:49:00.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.845+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:00.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f9e0 con 0x7f63e0064730 2021-03-09T08:49:00.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.846+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:00.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.846+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.846+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:00.859 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01f800 con 0x7f63e0064730 2021-03-09T08:49:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.847+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.847+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:49:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:00.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047780 con 0x7f63e0064730 2021-03-09T08:49:01.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:01.342+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:01.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:01.342+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:02.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:02.342+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:02.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:02.342+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:02.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:02.342+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:03.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:03.342+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:03.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:03.342+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:04.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:04.342+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:04.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:04.342+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:04.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:04.345+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:05.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.342+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:05.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.342+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:05.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.841+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:49:05.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.841+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.841+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002b580 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f63fc010e20 tx=0x7f63fc00baa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.841+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002b580 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f63fc010e20 tx=0x7f63fc00baa0).stop 2021-03-09T08:49:05.844 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:05.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:05.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:05.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002b580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:05.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002b580 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:05.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:49:05.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:05.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:05.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:05.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:05.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:05.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:05.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:05.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066da0 con 0x7f63e00070a0 2021-03-09T08:49:05.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:05.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c2a0 con 0x7f6400145bd0 2021-03-09T08:49:05.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:05.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000ef0 con 0x7f6400146050 2021-03-09T08:49:05.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:05.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:05.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:05.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:05.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.842+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:05.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:05.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.843+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:49:05.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.843+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:05.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.843+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:05.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:05.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:05.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:05.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:05.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:05.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:05.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:05.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:05.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:49:05.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:49:05.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:05.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:05.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d090 con 0x7f6400146050 2021-03-09T08:49:05.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:05.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:05.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63f00159e0 con 0x7f6400146050 2021-03-09T08:49:05.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:49:05.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.844+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:05.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.848+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:49:05.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f63f004d8d0 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.848+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002f070 con 0x7f6400146050 2021-03-09T08:49:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.853+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.853+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.853+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:05.861 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:05.861 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:05.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d940 con 0x7f6400146050 2021-03-09T08:49:05.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.853+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:05.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0006380 con 0x7f6400146050 2021-03-09T08:49:05.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:05.853+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:49:06.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:06.342+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:06.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:06.342+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:07.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:07.342+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:07.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:07.342+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:08.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:08.342+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:08.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:08.342+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:09.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:09.343+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:09.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:09.343+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:09.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:09.346+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:10.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.343+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:10.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.343+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:10.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0029950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:49:10.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0029950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:10.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f63f004d8d0 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029950 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f63f004d8d0 tx=0x7f63f004d180).stop 2021-03-09T08:49:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0029950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029950 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:49:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:10.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:10.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00263a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:10.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:10.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:10.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005b430 con 0x7f63e00070a0 2021-03-09T08:49:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002aed0 con 0x7f6400145bd0 2021-03-09T08:49:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0062020 con 0x7f63e0064730 2021-03-09T08:49:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:10.849 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:10.842+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.843+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:10.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.843+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:10.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00263a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:10.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.843+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:49:10.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.843+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:10.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.843+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:10.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:10.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:10.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:10.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:10.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:10.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:10.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:10.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:10.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:49:10.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:49:10.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:10.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026890 con 0x7f63e0064730 2021-03-09T08:49:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63f00159e0 con 0x7f63e0064730 2021-03-09T08:49:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:49:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:49:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00263a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f63f0015b70 tx=0x7f63f004e0c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0050a00 con 0x7f63e0064730 2021-03-09T08:49:10.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.846+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:10.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.846+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:10.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.846+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:10.858 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:10.860 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0050c10 con 0x7f63e0064730 2021-03-09T08:49:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.847+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.848+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:49:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:10.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00390a0 con 0x7f63e0064730 2021-03-09T08:49:11.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:11.343+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:11.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:11.343+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:12.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:12.343+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:12.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:12.343+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:12.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:12.343+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:13.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:13.343+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:13.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:13.343+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:14.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:14.343+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:14.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:14.343+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:14.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:14.347+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:15.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.343+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:15.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.343+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:15.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00263a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:49:15.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00263a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:15.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00263a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f63f0015b70 tx=0x7f63f004e0c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:15.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00263a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f63f0015b70 tx=0x7f63f004e0c0).stop 2021-03-09T08:49:15.846 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:15.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:15.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:15.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00263a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:15.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00263a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:15.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:49:15.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:15.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:15.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:15.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:15.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:15.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:15.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:15.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005b430 con 0x7f63e00070a0 2021-03-09T08:49:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000db0 con 0x7f6400145bd0 2021-03-09T08:49:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ad80 con 0x7f6400146050 2021-03-09T08:49:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.843+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.844+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.844+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:49:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.844+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.844+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:15.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:15.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:15.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:15.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:49:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:49:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0076dd0 con 0x7f6400146050 2021-03-09T08:49:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ac50 con 0x7f6400146050 2021-03-09T08:49:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:49:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.845+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:15.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.846+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:49:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.846+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.846+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f40058e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400be00 con 0x7f6400146050 2021-03-09T08:49:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.847+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.847+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.847+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:15.862 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:15.862 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:15.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.849+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003960 con 0x7f6400146050 2021-03-09T08:49:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.849+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.849+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:49:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:15.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400f030 con 0x7f6400146050 2021-03-09T08:49:16.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:16.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:16.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:16.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:17.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:17.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:17.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:17.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:18.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:18.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:18.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:18.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:19.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:19.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:19.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:19.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:19.348+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:20.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:20.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:20.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:49:20.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:20.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f40058e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:20.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025910 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f40058e0).stop 2021-03-09T08:49:20.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:20.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:20.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:20.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025910 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:20.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:49:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:20.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c150 con 0x7f63e00070a0 2021-03-09T08:49:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023240 con 0x7f6400145bd0 2021-03-09T08:49:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0062020 con 0x7f63e0064730 2021-03-09T08:49:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:20.851 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 70 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:49:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:49:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:49:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0069630 con 0x7f63e0064730 2021-03-09T08:49:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ac50 con 0x7f63e0064730 2021-03-09T08:49:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:49:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.845+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.846+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022d50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f63f400c010 tx=0x7f63f400b5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.846+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:49:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.846+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f63e0064730 2021-03-09T08:49:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.846+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.846+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.846+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:20.863 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4048e90 con 0x7f63e0064730 2021-03-09T08:49:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.847+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.847+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:49:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:20.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4026650 con 0x7f63e0064730 2021-03-09T08:49:21.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:21.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:21.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:21.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:22.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:22.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:22.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:22.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:22.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:22.344+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:23.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:23.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:23.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:23.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:24.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:24.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:24.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:24.345+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:24.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:24.349+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:25.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:25.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:25.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0022d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:49:25.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0022d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:25.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022d50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f63f400c010 tx=0x7f63f400b5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:25.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022d50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f63f400c010 tx=0x7f63f400b5d0).stop 2021-03-09T08:49:25.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:25.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:25.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0022d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:25.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022d50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:25.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:49:25.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:25.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:25.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:25.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:25.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:25.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:25.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:25.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027290 con 0x7f63e00070a0 2021-03-09T08:49:25.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:25.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000aaf0 con 0x7f6400145bd0 2021-03-09T08:49:25.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:25.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cf30 con 0x7f6400146050 2021-03-09T08:49:25.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.844+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:25.851 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:25.844+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 105 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:25.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:25.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:25.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.844+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:25.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.844+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:25.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:25.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.845+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:49:25.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.845+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:25.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.845+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:25.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:25.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:49:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:49:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e430 con 0x7f6400146050 2021-03-09T08:49:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ac50 con 0x7f6400146050 2021-03-09T08:49:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:49:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:49:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:25.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ec90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f63f404b090 tx=0x7f63f4046a60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:25.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404b100 con 0x7f6400146050 2021-03-09T08:49:25.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.848+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:25.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.848+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.848+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:25.863 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404b100 con 0x7f6400146050 2021-03-09T08:49:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.848+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.849+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:49:25.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:25.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:49:26.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:26.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:26.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:26.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:27.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:27.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:27.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:27.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:28.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:28.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:28.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:28.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:29.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:29.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:29.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:29.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:29.349+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:30.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:30.847 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 140 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:30.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:49:30.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:30.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ec90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f63f404b090 tx=0x7f63f4046a60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:30.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ec90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f63f404b090 tx=0x7f63f4046a60).stop 2021-03-09T08:49:30.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:30.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:30.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001ec90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:30.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ec90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:30.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:49:30.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:30.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:30.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:30.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:30.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:30.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000a9b0 con 0x7f63e00070a0 2021-03-09T08:49:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:30.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f6400145bd0 2021-03-09T08:49:30.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002020 con 0x7f63e0064730 2021-03-09T08:49:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:30.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:30.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.845+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.845+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:49:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.845+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.845+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.845+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.845+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.845+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:49:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:49:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00304b0 con 0x7f63e0064730 2021-03-09T08:49:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f640005e040 con 0x7f63e0064730 2021-03-09T08:49:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:49:30.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.846+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:30.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.847+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:49:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.847+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cd40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f63fc075b90 tx=0x7f63fc00b940).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.847+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047cc0 con 0x7f63e0064730 2021-03-09T08:49:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.848+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.848+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.848+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.849+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047af0 con 0x7f63e0064730 2021-03-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.849+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.849+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:49:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:30.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63e0064730 2021-03-09T08:49:31.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:31.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:31.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:31.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:32.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:32.344+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:32.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:32.344+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:32.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:32.344+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:33.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:33.345+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:33.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:33.345+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:34.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:34.345+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:34.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:34.345+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:34.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:34.350+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:35.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.345+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:35.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.345+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:35.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.844+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:49:35.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.844+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:35.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cd40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f63fc075b90 tx=0x7f63fc00b940).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:35.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cd40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f63fc075b90 tx=0x7f63fc00b940).stop 2021-03-09T08:49:35.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:35.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:35.849 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 172 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:35.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001cd40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:35.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cd40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:35.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:49:35.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:35.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:35.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:35.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:35.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00192c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:35.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:35.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:35.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0055960 con 0x7f63e00070a0 2021-03-09T08:49:35.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:35.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0003040 con 0x7f6400145bd0 2021-03-09T08:49:35.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:35.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00098f0 con 0x7f6400146050 2021-03-09T08:49:35.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:35.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:35.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:35.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:35.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:35.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.846+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:35.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.846+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:49:35.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.846+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:35.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.846+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:35.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:35.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:35.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:35.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:35.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:35.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:35.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:35.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:35.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:49:35.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:49:35.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:35.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:35.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000a960 con 0x7f6400146050 2021-03-09T08:49:35.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:35.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:35.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63f00159e0 con 0x7f6400146050 2021-03-09T08:49:35.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:49:35.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.847+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:35.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.848+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:49:35.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.848+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:35.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00192c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f63f00037d0 tx=0x7f63f0050910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:35.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003b4f0 con 0x7f6400146050 2021-03-09T08:49:35.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.850+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:35.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.850+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:35.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.850+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:35.863 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:35.863 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:35.863 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:35.865 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:35.865 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:35.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003b310 con 0x7f6400146050 2021-03-09T08:49:35.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.850+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:35.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.851+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:49:35.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:35.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f6400146050 2021-03-09T08:49:36.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:36.345+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:36.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:36.345+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:37.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:37.345+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:37.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:37.345+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:38.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:38.345+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:38.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:38.345+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:39.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:39.345+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:39.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:39.345+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:39.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:39.351+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:40.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.346+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:40.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.346+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:40.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00192c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:49:40.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00192c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:40.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00192c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f63f00037d0 tx=0x7f63f0050910).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:40.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00192c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f63f00037d0 tx=0x7f63f0050910).stop 2021-03-09T08:49:40.849 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 209 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:40.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:40.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:40.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00192c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:40.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00192c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:40.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:49:40.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:40.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:40.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00185d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:40.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:40.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:40.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:40.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:40.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e5d0 con 0x7f63e00070a0 2021-03-09T08:49:40.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:40.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00098f0 con 0x7f6400145bd0 2021-03-09T08:49:40.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:40.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009760 con 0x7f63e0064730 2021-03-09T08:49:40.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:40.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:40.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:40.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.846+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:40.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.846+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:40.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.846+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00185d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:40.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.846+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:49:40.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.846+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:40.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.846+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:40.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.847+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:40.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.847+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:40.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.847+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:40.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:40.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:40.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:40.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:40.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:40.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:49:40.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:49:40.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:40.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002abf0 con 0x7f63e0064730 2021-03-09T08:49:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00304b0 con 0x7f63e0064730 2021-03-09T08:49:40.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:49:40.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:40.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00185d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:40.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:49:40.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.848+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:40.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.849+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00de40 con 0x7f63e0064730 2021-03-09T08:49:40.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.849+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:40.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.849+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.849+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:40.865 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00dc60 con 0x7f63e0064730 2021-03-09T08:49:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.851+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.851+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:49:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:40.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00d7f0 con 0x7f63e0064730 2021-03-09T08:49:41.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:41.346+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:41.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:41.346+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:42.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:42.346+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:42.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:42.346+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:42.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:42.346+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:43.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:43.346+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:43.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:43.346+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:44.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:44.346+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:44.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:44.346+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:44.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:44.352+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:45.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.346+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:45.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.346+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:45.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00185d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:49:45.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00185d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:45.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00185d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:45.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00185d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009d60).stop 2021-03-09T08:49:45.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:45.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:45.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00185d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:45.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00185d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:45.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:49:45.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00155e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:45.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:45.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:45.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:45.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:45.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:45.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:45.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002ef0 con 0x7f63e00070a0 2021-03-09T08:49:45.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:45.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00098f0 con 0x7f6400145bd0 2021-03-09T08:49:45.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:45.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c50 con 0x7f6400146050 2021-03-09T08:49:45.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:45.854 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 242 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:45.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:45.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:45.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:45.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.846+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:45.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00155e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:45.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.847+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:49:45.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.847+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:45.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.847+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:45.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.847+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:45.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.847+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:45.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.847+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:45.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:45.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:45.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:45.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:45.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:45.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:49:45.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:49:45.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:45.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:45.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001fd0 con 0x7f6400146050 2021-03-09T08:49:45.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:45.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:45.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ac50 con 0x7f6400146050 2021-03-09T08:49:45.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:49:45.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.849+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:45.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.850+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:49:45.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.850+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00155e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4009230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:45.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.850+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:45.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c020 con 0x7f6400146050 2021-03-09T08:49:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.853+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.853+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.853+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:45.869 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:45.869 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:45.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404e380 con 0x7f6400146050 2021-03-09T08:49:45.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.853+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:45.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.853+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:49:45.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:45.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:49:46.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:46.346+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:46.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:46.346+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:47.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:47.347+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:47.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:47.347+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:48.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:48.346+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:48.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:48.346+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:49.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:49.347+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:49.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:49.347+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:49.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:49.353+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:50.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.347+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:50.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.347+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:50.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00155e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:49:50.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00155e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:50.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00155e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4009230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00155e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4009230).stop 2021-03-09T08:49:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00155e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:50.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00155e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:49:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:50.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:50.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00098f0 con 0x7f63e00070a0 2021-03-09T08:49:50.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:50.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c50 con 0x7f6400145bd0 2021-03-09T08:49:50.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:50.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a660 con 0x7f63e0064730 2021-03-09T08:49:50.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:50.854 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 279 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.846+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.847+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:49:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.847+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.847+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.847+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.847+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.847+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:49:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:49:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014950 con 0x7f63e0064730 2021-03-09T08:49:50.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:50.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:50.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63f00159e0 con 0x7f63e0064730 2021-03-09T08:49:50.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:49:50.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.848+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:50.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.849+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:49:50.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014460 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f63e003a0f0 tx=0x7f63f0006e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:50.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.849+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:50.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013be0 con 0x7f63e0064730 2021-03-09T08:49:50.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.851+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:50.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.851+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:50.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.851+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:50.864 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:50.864 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:50.864 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:50.866 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013be0 con 0x7f63e0064730 2021-03-09T08:49:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.854+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.854+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:49:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:50.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e0064730 2021-03-09T08:49:51.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:51.347+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:51.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:51.347+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:52.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:52.347+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:52.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:52.347+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:52.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:52.347+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:53.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:53.347+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:53.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:53.347+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:54.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:54.348+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:54.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:54.348+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:54.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:54.354+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:49:55.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.348+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:55.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.348+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:55.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0014460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:49:55.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0014460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:49:55.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014460 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f63e003a0f0 tx=0x7f63f0006e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:49:55.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014460 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f63e003a0f0 tx=0x7f63f0006e00).stop 2021-03-09T08:49:55.852 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 316 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:49:55.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:55.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:49:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0014460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:49:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014460 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:49:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:49:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:49:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:55.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:49:55.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:49:55.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:49:55.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:55.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004f9c0 con 0x7f63e00070a0 2021-03-09T08:49:55.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:55.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:49:55.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:49:55.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00098f0 con 0x7f6400146050 2021-03-09T08:49:55.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:49:55.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:49:55.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:55.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:49:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:49:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:49:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:49:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:49:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.847+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:49:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:49:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:49:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:49:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:49:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:55.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:55.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:49:55.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:49:55.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:49:55.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:49:55.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:49:55.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:55.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cda0 con 0x7f6400146050 2021-03-09T08:49:55.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:49:55.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:49:55.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63f00159e0 con 0x7f6400146050 2021-03-09T08:49:55.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.849+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:49:55.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.849+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:55.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.850+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011610 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0024b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:49:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.850+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:49:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.850+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:49:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00246f0 con 0x7f6400146050 2021-03-09T08:49:55.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.851+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:49:55.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.851+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:49:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.851+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:49:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:49:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:49:55.881 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:55.881 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:49:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:49:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:49:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:49:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:49:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:49:55.884 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:49:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00240a0 con 0x7f6400146050 2021-03-09T08:49:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.855+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:49:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.855+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:49:55.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:55.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T08:49:56.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:56.348+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:56.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:56.348+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:57.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:57.348+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:57.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:57.348+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:58.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:58.348+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:58.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:58.348+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:59.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:59.348+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:49:59.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:59.348+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:49:59.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:49:59.355+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:00.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.348+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:00.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.348+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:00.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0011610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:50:00.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0011610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:00.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011610 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0024b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:00.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011610 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0024b00).stop 2021-03-09T08:50:00.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:00.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:00.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0011610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:00.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011610 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:00.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:50:00.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:00.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:00.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:00.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:00.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:00.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:00.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:00.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:50:00.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:00.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a680 con 0x7f6400145bd0 2021-03-09T08:50:00.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:00.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009780 con 0x7f63e0064730 2021-03-09T08:50:00.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:00.855 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 357 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:00.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.848+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.848+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.848+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.848+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:50:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.848+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.848+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:00.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:00.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:00.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:00.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:00.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:50:00.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:50:00.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:00.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:00.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e2a0 con 0x7f63e0064730 2021-03-09T08:50:00.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:00.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:00.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ac50 con 0x7f63e0064730 2021-03-09T08:50:00.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:50:00.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.849+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:00.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.850+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:50:00.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.850+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f1d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f63f4007940 tx=0x7f63f40038a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:00.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.850+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:00.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c040 con 0x7f63e0064730 2021-03-09T08:50:00.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.852+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:00.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.852+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:00.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.852+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:00.865 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:00.866 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:00.866 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:00.866 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:00.866 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:00.866 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:00.867 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:00.867 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:00.867 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:00.867 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4007740 con 0x7f63e0064730 2021-03-09T08:50:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.853+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.854+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:50:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:00.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T08:50:01.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:01.348+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:01.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:01.348+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:02.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:02.348+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:02.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:02.348+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:02.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:02.348+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:03.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:03.349+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:03.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:03.349+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:04.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:04.349+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:04.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:04.349+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:04.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:04.356+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:05.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.349+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:05.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.349+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:05.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:50:05.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:05.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f1d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f63f4007940 tx=0x7f63f40038a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:05.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f1d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f63f4007940 tx=0x7f63f40038a0).stop 2021-03-09T08:50:05.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:05.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:05.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000f1d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:05.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f1d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:05.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:50:05.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:05.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:05.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:05.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:05.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:05.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:05.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:05.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004f9c0 con 0x7f63e00070a0 2021-03-09T08:50:05.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:50:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f6400146050 2021-03-09T08:50:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:05.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:05.861 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:05.848+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 395 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:05.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.849+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:05.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.849+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:50:05.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.849+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:05.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.849+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:05.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.849+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.849+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.849+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:50:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:50:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001db0 con 0x7f6400146050 2021-03-09T08:50:05.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:05.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:05.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002abf0 con 0x7f6400146050 2021-03-09T08:50:05.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:50:05.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.850+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:05.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.851+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:50:05.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.851+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000cf50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc071630).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047cc0 con 0x7f6400146050 2021-03-09T08:50:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.852+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.852+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.852+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:05.882 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047cc0 con 0x7f6400146050 2021-03-09T08:50:05.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.852+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:05.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.853+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:50:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:05.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f6400146050 2021-03-09T08:50:06.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:06.349+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:06.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:06.349+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:07.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:07.349+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:07.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:07.349+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:08.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:08.349+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:08.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:08.349+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:09.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:09.349+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:09.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:09.349+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:09.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:09.357+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:10.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.349+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:10.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.349+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:10.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:50:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000cf50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc071630).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000cf50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc071630).stop 2021-03-09T08:50:10.852 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 433 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:10.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:10.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:10.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000cf50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:10.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000cf50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:10.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:50:10.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:10.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:10.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:10.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:10.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:10.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:10.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:50:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004f9c0 con 0x7f6400145bd0 2021-03-09T08:50:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:10.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027140 con 0x7f63e0064730 2021-03-09T08:50:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.848+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.849+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.849+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:10.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.849+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:10.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:10.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.849+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:50:10.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.849+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:10.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.849+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.850+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.850+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.850+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:50:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:50:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:50:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63f00159e0 con 0x7f63e0064730 2021-03-09T08:50:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:50:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:50:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0078f70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f63f0024210 tx=0x7f63f001f510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d710 con 0x7f63e0064730 2021-03-09T08:50:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.855+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.855+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.855+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:10.867 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:10.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0064730 2021-03-09T08:50:10.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.855+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:10.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.855+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:50:10.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:10.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:50:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:11.349+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:11.349+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:12.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:12.350+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:12.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:12.350+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:12.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:12.350+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:13.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:13.350+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:13.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:13.350+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:14.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:14.350+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:14.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:14.350+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:14.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:14.358+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:15.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.350+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:15.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.350+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:50:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0078f70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f63f0024210 tx=0x7f63f001f510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0078f70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f63f0024210 tx=0x7f63f001f510).stop 2021-03-09T08:50:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:50:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:15.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:15.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:15.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:15.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004f9c0 con 0x7f63e00070a0 2021-03-09T08:50:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f6400145bd0 2021-03-09T08:50:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e0d0 con 0x7f6400146050 2021-03-09T08:50:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:15.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:15.858 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 470 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:15.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:15.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.850+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:50:15.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.850+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:15.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.850+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.850+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.850+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.850+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:15.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:15.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:15.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:15.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:50:15.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:50:15.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:15.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:15.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015ec0 con 0x7f6400146050 2021-03-09T08:50:15.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:15.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:15.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001db0 con 0x7f6400146050 2021-03-09T08:50:15.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:50:15.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:15.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:50:15.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:15.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc00a8a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:15.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01de70 con 0x7f6400146050 2021-03-09T08:50:15.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.852+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:15.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.852+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:15.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.852+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:15.868 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:15.870 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T08:50:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.852+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.852+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:50:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:15.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f6400146050 2021-03-09T08:50:16.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:16.350+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:16.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:16.350+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:17.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:17.350+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:17.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:17.350+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:18.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:18.351+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:18.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:18.351+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:19.351+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:19.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:19.351+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:19.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:19.359+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:20.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.351+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:20.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.351+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:50:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc00a8a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc00a8a0).stop 2021-03-09T08:50:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000d3a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:50:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:20.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0044650 con 0x7f63e00070a0 2021-03-09T08:50:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:50:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004a420 con 0x7f63e0064730 2021-03-09T08:50:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:20.859 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 505 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:50:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.850+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:50:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:50:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001dcf0 con 0x7f63e0064730 2021-03-09T08:50:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ac50 con 0x7f63e0064730 2021-03-09T08:50:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:50:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-03-09T08:50:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2021-03-09T08:50:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0x7f63f4000e50 tx=0x7f63f40470f0).write client ident write failed r=-32 ((32) Broken pipe) 2021-03-09T08:50:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:20.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:21.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6405551700 10 monclient: get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:50:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6405551700 0 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-03-09T08:50:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001d800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:50:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0050df0 con 0x7f63e00070a0 2021-03-09T08:50:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:50:21.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:21.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004a420 con 0x7f6400146050 2021-03-09T08:50:21.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:21.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:21.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:21.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:21.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:21.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:50:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:50:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:50:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001de80 con 0x7f6400146050 2021-03-09T08:50:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015ec0 con 0x7f6400146050 2021-03-09T08:50:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:50:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:50:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.052+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0470c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.053+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T08:50:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.053+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.053+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.053+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:21.071 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:21.071 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:21.071 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:21.071 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:21.073 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05aa50 con 0x7f6400146050 2021-03-09T08:50:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.053+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.053+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:50:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00abd0 con 0x7f6400146050 2021-03-09T08:50:21.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.351+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:21.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:21.351+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:22.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:22.351+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:22.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:22.351+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:22.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:22.351+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:23.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:23.351+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:23.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:23.351+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:24.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:24.351+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:24.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:24.351+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:24.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:24.360+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:25.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.351+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:25.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.351+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:25.853 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:25.850+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 540 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:25.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:50:25.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:25.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0470c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:25.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0470c0).stop 2021-03-09T08:50:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.850+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000d3a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000d3a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:50:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0044650 con 0x7f63e00070a0 2021-03-09T08:50:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:50:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004a420 con 0x7f63e0064730 2021-03-09T08:50:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:25.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:25.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:25.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:50:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.851+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:50:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:50:25.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:25.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:25.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0032660 con 0x7f63e0064730 2021-03-09T08:50:25.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:25.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:25.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001dcf0 con 0x7f63e0064730 2021-03-09T08:50:25.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:50:25.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:25.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:50:25.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:25.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.852+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f63f403f070 tx=0x7f63f4005ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:25.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f63e0064730 2021-03-09T08:50:25.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.854+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:25.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.854+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:25.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.854+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:25.868 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:25.868 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:25.868 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:25.868 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:25.868 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:25.869 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40474c0 con 0x7f63e0064730 2021-03-09T08:50:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.854+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.854+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:50:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:25.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T08:50:26.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:26.352+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:26.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:26.352+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:27.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:27.351+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:27.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:27.351+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:28.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:28.352+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:28.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:28.352+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:29.352+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:29.352+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:29.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:29.361+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:30.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.352+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:30.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.352+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:30.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0029910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:50:30.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0029910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:30.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f63f403f070 tx=0x7f63f4005ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:30.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f63f403f070 tx=0x7f63f4005ea0).stop 2021-03-09T08:50:30.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:30.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:30.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0029910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:30.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029910 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:30.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:50:30.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:30.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:30.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:30.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:30.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:30.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:50:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00792e0 con 0x7f6400145bd0 2021-03-09T08:50:30.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:30.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0038940 con 0x7f6400146050 2021-03-09T08:50:30.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:30.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:30.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:30.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:50:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:30.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:30.860 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:30.851+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 578 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:30.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:50:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:50:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025ca0 con 0x7f6400146050 2021-03-09T08:50:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0032660 con 0x7f6400146050 2021-03-09T08:50:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:50:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.852+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.853+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:50:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.853+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.853+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f63f400ab30 tx=0x7f63f4014fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022ec0 con 0x7f6400146050 2021-03-09T08:50:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.855+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.855+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.855+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:30.869 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f6400146050 2021-03-09T08:50:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.855+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:50:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:30.856+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:50:31.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:31.352+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:31.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:31.352+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:32.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:32.352+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:32.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:32.352+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:32.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:32.352+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:33.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:33.353+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:33.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:33.353+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:34.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:34.353+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:34.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:34.353+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:34.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:34.362+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:35.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.353+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:35.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.353+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:35.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:50:35.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:35.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f63f400ab30 tx=0x7f63f4014fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:35.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f63f400ab30 tx=0x7f63f4014fb0).stop 2021-03-09T08:50:35.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:35.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:35.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:35.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:35.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:50:35.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:35.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:35.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:35.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:35.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:35.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:35.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:35.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0049be0 con 0x7f63e00070a0 2021-03-09T08:50:35.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:35.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004a420 con 0x7f6400145bd0 2021-03-09T08:50:35.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:35.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0044650 con 0x7f63e0064730 2021-03-09T08:50:35.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:35.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.852+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:35.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.852+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:35.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.852+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:35.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.852+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:35.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.852+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.852+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:50:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.852+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:35.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.852+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:35.865 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:35.851+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 618 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:35.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:35.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:35.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:35.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:50:35.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:50:35.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:35.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:35.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015ec0 con 0x7f63e0064730 2021-03-09T08:50:35.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:35.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025ca0 con 0x7f63e0064730 2021-03-09T08:50:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.854+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:50:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.854+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.854+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:50:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.854+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.854+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f63f4028420 tx=0x7f63f403c860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f6405551700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T08:50:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:50:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f63f4028420 tx=0x7f63f403c860).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f63f4028420 tx=0x7f63f403c860).stop 2021-03-09T08:50:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002a200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:50:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:50:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0044650 con 0x7f6400145bd0 2021-03-09T08:50:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0049be0 con 0x7f6400146050 2021-03-09T08:50:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:50:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:50:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:50:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0032660 con 0x7f6400146050 2021-03-09T08:50:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015ec0 con 0x7f6400146050 2021-03-09T08:50:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:50:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.857+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.858+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:50:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.858+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f63f403f070 tx=0x7f63f4044f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.858+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f6400146050 2021-03-09T08:50:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.859+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.859+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.859+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:35.886 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:35.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40224e0 con 0x7f6400146050 2021-03-09T08:50:35.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.860+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:35.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.860+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:50:35.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:35.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:50:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:36.353+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:36.353+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:37.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:37.353+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:37.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:37.353+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:38.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:38.353+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:38.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:38.353+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:39.353+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:39.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:39.353+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:39.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:39.363+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:40.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.354+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:40.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.354+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:40.855 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 656 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:40.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:50:40.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:40.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f63f403f070 tx=0x7f63f4044f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:40.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f63f403f070 tx=0x7f63f4044f20).stop 2021-03-09T08:50:40.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:40.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:40.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:40.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017f70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:40.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:50:40.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:40.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:40.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:40.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:40.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004a420 con 0x7f63e00070a0 2021-03-09T08:50:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:40.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017900 con 0x7f6400145bd0 2021-03-09T08:50:40.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:40.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0044650 con 0x7f63e0064730 2021-03-09T08:50:40.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.852+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:40.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.853+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:40.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.853+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:40.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:40.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.853+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:40.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:40.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.854+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:50:40.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.854+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:40.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.854+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.854+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.854+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.854+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:50:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:50:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bfc0 con 0x7f63e0064730 2021-03-09T08:50:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001de80 con 0x7f63e0064730 2021-03-09T08:50:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:50:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:50:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f63e0012340 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.855+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01de70 con 0x7f63e0064730 2021-03-09T08:50:40.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.856+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:40.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.856+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:40.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.856+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:40.868 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:40.869 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:40.869 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:40.869 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:40.869 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:40.870 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:40.870 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:40.870 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:40.870 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:40.871 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:40.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T08:50:40.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.857+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:40.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.857+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:50:40.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:40.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0064730 2021-03-09T08:50:41.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:41.354+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:41.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:41.354+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:42.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:42.354+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:42.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:42.354+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:42.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:42.354+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:43.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:43.354+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:43.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:43.354+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:44.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:44.354+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:44.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:44.354+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:44.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:44.364+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.354+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.354+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:45.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:50:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f63e0012340 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:45.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f63e0012340 tx=0x7f63fc04a740).stop 2021-03-09T08:50:45.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:45.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:45.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002a200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:45.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002a200 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:45.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:50:45.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:45.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:45.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:45.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:45.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:45.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:45.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:45.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a680 con 0x7f63e00070a0 2021-03-09T08:50:45.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:45.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001dd40 con 0x7f6400145bd0 2021-03-09T08:50:45.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:45.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0049be0 con 0x7f6400146050 2021-03-09T08:50:45.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:45.862 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 691 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:50:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.853+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.854+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.854+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.854+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:45.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:45.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:45.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:45.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:50:45.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:50:45.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:45.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:45.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d50 con 0x7f6400146050 2021-03-09T08:50:45.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:45.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:45.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bfc0 con 0x7f6400146050 2021-03-09T08:50:45.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:50:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:50:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a2f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f63e000f0f0 tx=0x7f63fc075120).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.855+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058810 con 0x7f6400146050 2021-03-09T08:50:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.857+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.857+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.857+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:45.877 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:45.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058810 con 0x7f6400146050 2021-03-09T08:50:45.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.860+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:45.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.860+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:50:45.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:45.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04abe0 con 0x7f6400146050 2021-03-09T08:50:46.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:46.354+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:46.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:46.354+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:47.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:47.354+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:47.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:47.354+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:48.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:48.355+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:48.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:48.355+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:49.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:49.355+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:49.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:49.355+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:49.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:49.365+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:50.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.355+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:50.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.355+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:50.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:50:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:50.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a2f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f63e000f0f0 tx=0x7f63fc075120).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a2f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f63e000f0f0 tx=0x7f63fc075120).stop 2021-03-09T08:50:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007a2f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a2f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:50:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:50.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:50.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:50:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004a420 con 0x7f6400145bd0 2021-03-09T08:50:50.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:50.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0049be0 con 0x7f63e0064730 2021-03-09T08:50:50.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:50.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:50.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:50.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:50.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:50.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:50.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:50:50.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:50.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:50.863 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:50.853+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 730 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.854+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:50:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:50:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0038940 con 0x7f63e0064730 2021-03-09T08:50:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d50 con 0x7f63e0064730 2021-03-09T08:50:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:50:50.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:50.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:50:50.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028e60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f63e00154d0 tx=0x7f63fc058cd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:50.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.855+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00e470 con 0x7f63e0064730 2021-03-09T08:50:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.857+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.857+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.857+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:50.871 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:50.871 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:50.871 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:50.871 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:50.872 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:50.872 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:50.872 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:50.872 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:50.872 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:50.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058810 con 0x7f63e0064730 2021-03-09T08:50:50.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.858+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:50.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.858+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:50:50.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:50.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00ed40 con 0x7f63e0064730 2021-03-09T08:50:51.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:51.355+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:51.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:51.355+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:52.355+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:52.355+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:52.355+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:53.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:53.355+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:53.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:53.355+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:54.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:54.355+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:54.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:54.355+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:54.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:54.366+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:50:55.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:55.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:55.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.853+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0028e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:50:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0028e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:50:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028e60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f63e00154d0 tx=0x7f63fc058cd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:50:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028e60 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f63e00154d0 tx=0x7f63fc058cd0).stop 2021-03-09T08:50:55.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:50:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0028e60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:50:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028e60 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:50:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:50:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:50:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:50:55.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:50:55.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:50:55.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:55.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b0d0 con 0x7f63e00070a0 2021-03-09T08:50:55.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:55.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0044650 con 0x7f6400145bd0 2021-03-09T08:50:55.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:50:55.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004a420 con 0x7f6400146050 2021-03-09T08:50:55.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:50:55.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:50:55.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:55.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:50:55.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:50:55.863 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:50:55.854+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 768 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:50:55.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:50:55.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.855+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:50:55.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.855+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:50:55.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.855+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:50:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.855+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:50:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.855+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:50:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.855+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:50:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:50:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:50:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:50:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:50:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:50:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:50:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e003b7e0 con 0x7f6400146050 2021-03-09T08:50:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:50:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:50:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e004deb0 con 0x7f6400146050 2021-03-09T08:50:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:50:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078f70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f0013bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:50:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:50:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.856+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:50:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T08:50:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.858+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:50:55.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.858+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:50:55.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.858+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:50:55.871 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:50:55.871 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:50:55.872 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:55.872 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:50:55.872 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:50:55.872 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:50:55.872 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:50:55.873 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:50:55.873 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:50:55.873 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:50:55.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d710 con 0x7f6400146050 2021-03-09T08:50:55.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.860+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:50:55.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.860+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:50:55.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:55.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T08:50:56.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:56.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:56.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:56.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:57.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:57.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:57.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:57.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:58.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:58.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:58.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:58.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:59.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:59.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:50:59.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:59.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:50:59.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:50:59.367+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:00.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:00.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:00.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:51:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078f70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f0013bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078f70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f0013bf0).stop 2021-03-09T08:51:00.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:51:00.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:00.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:00.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:00.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:00.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:00.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:00.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:00.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f63e00070a0 2021-03-09T08:51:00.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:00.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011da0 con 0x7f6400145bd0 2021-03-09T08:51:00.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:00.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023450 con 0x7f63e0064730 2021-03-09T08:51:00.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:00.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:00.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:00.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:00.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:00.865 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:00.855+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 804 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:00.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:00.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:51:00.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:00.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:00.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:00.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:00.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:00.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:00.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:51:00.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:51:00.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:00.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:00.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e003d770 con 0x7f63e0064730 2021-03-09T08:51:00.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:00.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:00.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e003b7e0 con 0x7f63e0064730 2021-03-09T08:51:00.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:51:00.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.856+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:00.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.857+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:51:00.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.857+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:00.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.857+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ca40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f63f0006c40 tx=0x7f63f004dcf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:00.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0015a20 con 0x7f63e0064730 2021-03-09T08:51:00.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.861+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:00.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.861+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:00.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.861+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:00.874 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:00.874 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:00.875 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:00.875 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:00.875 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:00.875 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:00.877 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:00.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0015a20 con 0x7f63e0064730 2021-03-09T08:51:00.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.864+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:00.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.864+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:51:00.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:00.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:51:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:01.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:01.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:01.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:02.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:02.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:02.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:02.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:02.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:02.356+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:03.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:03.357+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:03.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:03.357+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:04.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:04.357+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:04.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:04.357+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:04.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:04.368+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:05.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:05.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:51:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:05.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ca40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f63f0006c40 tx=0x7f63f004dcf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ca40 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f63f0006c40 tx=0x7f63f004dcf0).stop 2021-03-09T08:51:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:05.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001ca40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ca40 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:51:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:05.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:05.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:05.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:05.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:05.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:05.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:05.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0049be0 con 0x7f63e00070a0 2021-03-09T08:51:05.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:05.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:51:05.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:05.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0044650 con 0x7f6400146050 2021-03-09T08:51:05.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:05.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:05.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:05.864 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:05.855+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 845 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:05.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:05.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:05.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:05.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:51:05.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:05.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:05.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:05.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.856+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:51:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:51:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e004a420 con 0x7f6400146050 2021-03-09T08:51:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0032660 con 0x7f6400146050 2021-03-09T08:51:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:51:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:51:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014420 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f63f403d560 tx=0x7f63f40037c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.857+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a0a0 con 0x7f6400146050 2021-03-09T08:51:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.858+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.858+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.858+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:05.873 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:05.873 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:05.873 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:05.873 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:05.873 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:05.874 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:05.874 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:05.874 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:05.874 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:05.875 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:05.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f403d170 con 0x7f6400146050 2021-03-09T08:51:05.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.858+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:05.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.858+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:51:05.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:05.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037080 con 0x7f6400146050 2021-03-09T08:51:06.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:06.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:06.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:06.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:07.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:07.356+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:07.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:07.356+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:08.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:08.357+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:08.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:08.357+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:09.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:09.357+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:09.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:09.357+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:09.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:09.369+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:10.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.358+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:10.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.358+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:51:10.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014420 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f63f403d560 tx=0x7f63f40037c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014420 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f63f403d560 tx=0x7f63f40037c0).stop 2021-03-09T08:51:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:10.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014420 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:51:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:10.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079320 con 0x7f63e00070a0 2021-03-09T08:51:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020890 con 0x7f6400145bd0 2021-03-09T08:51:10.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023450 con 0x7f63e0064730 2021-03-09T08:51:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:10.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:51:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:10.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:10.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:10.867 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:10.856+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 879 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:10.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:10.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:10.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:10.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:51:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:51:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bfc0 con 0x7f63e0064730 2021-03-09T08:51:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e004a420 con 0x7f63e0064730 2021-03-09T08:51:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:51:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.857+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.858+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:51:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.858+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.858+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f63f403f070 tx=0x7f63f400c380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a040 con 0x7f63e0064730 2021-03-09T08:51:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.860+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.860+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.860+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:10.875 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:10.875 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:10.875 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:10.875 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:10.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63e0064730 2021-03-09T08:51:10.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.862+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:10.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.862+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:51:10.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:10.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037030 con 0x7f63e0064730 2021-03-09T08:51:11.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:11.358+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:11.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:11.358+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:12.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:12.358+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:12.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:12.358+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:12.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:12.358+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:13.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:13.358+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:13.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:13.358+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:14.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:14.358+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:14.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:14.358+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:14.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:14.370+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:15.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.358+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:15.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.358+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.856+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:51:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.856+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.856+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f63f403f070 tx=0x7f63f400c380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.856+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f63f403f070 tx=0x7f63f400c380).stop 2021-03-09T08:51:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:15.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:51:15.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:15.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:15.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:15.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:15.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f63e00070a0 2021-03-09T08:51:15.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:15.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011da0 con 0x7f6400145bd0 2021-03-09T08:51:15.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:15.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:51:15.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:15.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:15.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:15.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:15.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:15.866 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:15.857+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 916 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.858+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:51:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.858+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.858+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.858+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.858+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.858+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:15.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:51:15.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:51:15.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:15.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:15.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e004deb0 con 0x7f6400146050 2021-03-09T08:51:15.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:15.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:15.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0038940 con 0x7f6400146050 2021-03-09T08:51:15.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:51:15.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.859+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:15.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.860+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:51:15.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.860+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:15.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.860+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc058d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:15.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc006ab0 con 0x7f6400146050 2021-03-09T08:51:15.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.861+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:15.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.861+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.861+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:15.877 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:15.877 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:15.877 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:15.877 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:15.878 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T08:51:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.862+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.862+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:51:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:15.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f6400146050 2021-03-09T08:51:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:16.358+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:16.358+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:17.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:17.358+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:17.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:17.358+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:18.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:18.359+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:18.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:18.359+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:19.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:19.359+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:19.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:19.359+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:19.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:19.371+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:20.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.359+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:20.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.359+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:51:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc058d40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc058d40).stop 2021-03-09T08:51:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:51:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079320 con 0x7f63e00070a0 2021-03-09T08:51:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016110 con 0x7f6400145bd0 2021-03-09T08:51:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018e10 con 0x7f63e0064730 2021-03-09T08:51:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:20.875 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 952 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:51:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.858+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:51:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:51:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e003b7e0 con 0x7f63e0064730 2021-03-09T08:51:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e003d770 con 0x7f63e0064730 2021-03-09T08:51:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:51:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.860+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.861+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:51:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.861+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.861+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f63e0011fc0 tx=0x7f63f00068a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e0064730 2021-03-09T08:51:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.862+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.862+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.862+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:20.885 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63e0064730 2021-03-09T08:51:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.864+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.864+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:51:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:20.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f63e0064730 2021-03-09T08:51:21.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:21.359+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:21.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:21.359+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:22.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:22.358+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:22.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:22.358+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:22.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:22.358+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:23.359+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:23.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:23.359+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:24.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:24.359+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:24.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:24.359+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:24.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:24.372+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:25.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.359+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:25.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.359+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:25.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:51:25.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:25.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f63e0011fc0 tx=0x7f63f00068a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:25.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f63e0011fc0 tx=0x7f63f00068a0).stop 2021-03-09T08:51:25.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b470 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:51:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f63e00070a0 2021-03-09T08:51:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018e10 con 0x7f6400145bd0 2021-03-09T08:51:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016110 con 0x7f6400146050 2021-03-09T08:51:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:25.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:25.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:25.873 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 994 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:25.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:51:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.859+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.860+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:25.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.860+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:25.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.860+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:25.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.860+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:25.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:25.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:25.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:25.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:25.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:51:25.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:51:25.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:25.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:25.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cc70 con 0x7f6400146050 2021-03-09T08:51:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e003b7e0 con 0x7f6400146050 2021-03-09T08:51:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:51:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.861+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.862+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:51:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.862+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f63f00133e0 tx=0x7f63f0015530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0019350 con 0x7f6400146050 2021-03-09T08:51:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.863+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.863+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.863+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:25.884 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0019350 con 0x7f6400146050 2021-03-09T08:51:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.866+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.866+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:51:25.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:25.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T08:51:26.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:26.360+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:26.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:26.360+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:27.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:27.360+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:27.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:27.360+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:28.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:28.359+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:28.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:28.359+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:29.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:29.360+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:29.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:29.360+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:29.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:29.373+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:30.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.360+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:30.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.360+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:51:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f63f00133e0 tx=0x7f63f0015530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:30.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7f63f00133e0 tx=0x7f63f0015530).stop 2021-03-09T08:51:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:30.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:51:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015130 con 0x7f63e00070a0 2021-03-09T08:51:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:51:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f63e0064730 2021-03-09T08:51:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:30.867 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:30.859+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1035 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.860+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.860+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:51:30.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.860+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:30.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.860+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.860+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.860+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.860+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:51:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:51:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026700 con 0x7f63e0064730 2021-03-09T08:51:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bfc0 con 0x7f63e0064730 2021-03-09T08:51:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:51:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.862+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.863+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:51:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.863+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f63f400c350 tx=0x7f63f4003650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.863+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f63e0064730 2021-03-09T08:51:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.863+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.863+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.863+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:30.877 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:30.877 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:30.877 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:30.877 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:30.878 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:30.878 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:30.878 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:30.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009900 con 0x7f63e0064730 2021-03-09T08:51:30.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.865+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:30.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.865+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:51:30.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:30.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T08:51:31.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:31.360+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:31.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:31.360+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:32.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:32.360+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:32.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:32.360+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:32.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:32.360+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:33.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:33.361+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:33.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:33.361+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:34.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:34.360+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:34.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:34.360+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:34.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:34.374+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:35.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.361+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:35.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.361+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:35.862 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1075 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:35.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:51:35.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f63f400c350 tx=0x7f63f4003650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f63f400c350 tx=0x7f63f4003650).stop 2021-03-09T08:51:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:35.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:35.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:35.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:51:35.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:35.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:35.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:35.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:35.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:35.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:35.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:35.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0035370 con 0x7f63e00070a0 2021-03-09T08:51:35.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:35.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016110 con 0x7f6400145bd0 2021-03-09T08:51:35.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:35.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:51:35.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:35.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:35.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:35.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.860+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.861+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.861+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:51:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.861+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:35.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.861+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:35.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.861+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:35.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.861+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:35.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.861+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:35.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:51:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:51:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bfc0 con 0x7f6400146050 2021-03-09T08:51:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cc70 con 0x7f6400146050 2021-03-09T08:51:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:51:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f63e002b030 tx=0x7f63f004ddd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:51:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.862+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004c0f0 con 0x7f6400146050 2021-03-09T08:51:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.864+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.864+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.864+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:35.879 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0046c80 con 0x7f6400146050 2021-03-09T08:51:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.865+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.865+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:51:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:35.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T08:51:36.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:36.361+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:36.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:36.361+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:37.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:37.361+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:37.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:37.361+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:38.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:38.360+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:38.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:38.360+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:39.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:39.361+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:39.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:39.361+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:39.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:39.375+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:40.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.361+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:40.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.361+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:51:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:40.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f63e002b030 tx=0x7f63f004ddd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f63e002b030 tx=0x7f63f004ddd0).stop 2021-03-09T08:51:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:40.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025750 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:51:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:40.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:40.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027000 con 0x7f63e00070a0 2021-03-09T08:51:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028770 con 0x7f6400145bd0 2021-03-09T08:51:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f63e0064730 2021-03-09T08:51:40.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.860+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:40.868 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:40.861+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1104 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:40.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.861+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:40.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.861+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:40.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.861+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:40.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.861+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:40.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.861+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:40.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.861+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:51:40.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.861+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:40.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.861+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:40.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.862+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:40.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.862+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:40.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.862+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:40.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.862+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:40.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.862+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:40.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:40.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:40.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:40.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:51:40.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:51:40.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:40.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:40.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015f00 con 0x7f63e0064730 2021-03-09T08:51:40.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:40.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:40.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:51:40.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:51:40.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc006d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:51:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.863+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc009cf0 con 0x7f63e0064730 2021-03-09T08:51:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.865+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.865+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.865+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:40.878 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:40.878 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:40.878 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:40.878 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:40.878 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:40.879 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:40.879 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:40.879 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:40.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e0064730 2021-03-09T08:51:40.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.869+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:40.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.869+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:51:40.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:40.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T08:51:41.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:41.362+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:41.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:41.362+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:42.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:42.362+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:42.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:42.362+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:42.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:42.362+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:43.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:43.362+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:43.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:43.362+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:44.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:44.362+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:44.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:44.362+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:44.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:44.376+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:45.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.361+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:45.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.361+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:45.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:51:45.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:45.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc006d40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc006d40).stop 2021-03-09T08:51:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:45.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bab0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:51:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:45.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:45.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015130 con 0x7f63e00070a0 2021-03-09T08:51:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016110 con 0x7f6400145bd0 2021-03-09T08:51:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:45.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000eb20 con 0x7f6400146050 2021-03-09T08:51:45.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:45.869 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1146 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:45.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:45.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:45.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:45.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:45.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:45.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:51:45.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.862+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:51:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:51:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015f00 con 0x7f6400146050 2021-03-09T08:51:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bfc0 con 0x7f6400146050 2021-03-09T08:51:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:51:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.863+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:45.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.864+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023e90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f0046f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:45.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.864+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:51:45.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.864+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:45.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005b100 con 0x7f6400146050 2021-03-09T08:51:45.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.866+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:45.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.866+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:45.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.866+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:45.879 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:45.880 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:45.880 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:45.880 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:45.880 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:45.880 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:45.881 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:45.881 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:45.881 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:45.881 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:45.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T08:51:45.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.866+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:45.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.866+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:51:45.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:45.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T08:51:46.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:46.362+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:46.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:46.362+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:47.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:47.362+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:47.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:47.362+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:48.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:48.363+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:48.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:48.363+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:49.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:49.363+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:49.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:49.363+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:49.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:49.377+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:50.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.363+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:50.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.363+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:50.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:51:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023e90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f0046f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:50.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023e90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f0046f20).stop 2021-03-09T08:51:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:50.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023e90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:51:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:50.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:50.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014ea0 con 0x7f63e00070a0 2021-03-09T08:51:50.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:50.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0033960 con 0x7f6400145bd0 2021-03-09T08:51:50.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:50.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0046f30 con 0x7f63e0064730 2021-03-09T08:51:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:50.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:50.871 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:50.862+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1183 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:50.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:50.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.863+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:51:50.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.863+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:50.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.863+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:50.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:50.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:50.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:50.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:50.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:50.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:50.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:50.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:50.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:51:50.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:51:50.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:50.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:50.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a8f0 con 0x7f63e0064730 2021-03-09T08:51:50.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:50.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:50.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:51:50.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:51:50.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.864+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:50.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.865+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f63e00273f0 tx=0x7f63fc01a380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:50.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.865+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:51:50.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.865+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:50.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T08:51:50.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.865+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:50.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.865+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:50.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.865+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:50.880 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:50.880 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:50.881 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:50.881 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:50.881 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:50.881 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:50.881 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:50.882 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:50.882 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:50.882 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:50.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc004e90 con 0x7f63e0064730 2021-03-09T08:51:50.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.868+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:50.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.868+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:51:50.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:50.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc019040 con 0x7f63e0064730 2021-03-09T08:51:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:51.363+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:51.363+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:52.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:52.363+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:52.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:52.363+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:52.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:52.363+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:53.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:53.363+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:53.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:53.363+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:54.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:54.364+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:54.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:54.364+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:54.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:54.378+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:51:55.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.364+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:55.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.364+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:55.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:51:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:51:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f63e00273f0 tx=0x7f63fc01a380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:51:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 rx=0x7f63e00273f0 tx=0x7f63fc01a380).stop 2021-03-09T08:51:55.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:51:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:51:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b430 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:51:55.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:51:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:51:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:51:55.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:51:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:51:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f63e00070a0 2021-03-09T08:51:55.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011160 con 0x7f6400145bd0 2021-03-09T08:51:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:51:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:51:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:51:55.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:51:55.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:55.871 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1222 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:51:55.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:51:55.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:51:55.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.864+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:51:55.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.864+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:51:55.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.864+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:51:55.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.864+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:51:55.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.864+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:51:55.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.864+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:51:55.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.864+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:51:55.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:51:55.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:55.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:55.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:51:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:51:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:51:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:51:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:51:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:55.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e004deb0 con 0x7f6400146050 2021-03-09T08:51:55.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:51:55.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:51:55.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026700 con 0x7f6400146050 2021-03-09T08:51:55.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:51:55.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.865+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:55.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.866+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f63f40073c0 tx=0x7f63f4009a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:51:55.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.866+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:51:55.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.866+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:51:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a040 con 0x7f6400146050 2021-03-09T08:51:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.867+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:51:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.867+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:51:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.867+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:51:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:51:55.881 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:51:55.881 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:55.881 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:51:55.881 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:51:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:51:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:51:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:51:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:51:55.882 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:51:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4047980 con 0x7f6400146050 2021-03-09T08:51:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.868+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:51:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.868+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:51:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:55.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:51:56.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:56.364+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:56.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:56.364+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:57.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:57.364+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:57.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:57.364+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:58.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:58.363+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:58.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:58.363+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:59.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:59.364+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:51:59.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:59.364+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:51:59.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:51:59.379+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:00.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.364+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:00.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.364+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:00.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.864+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:52:00.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.864+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:00.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.864+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f63f40073c0 tx=0x7f63f4009a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.864+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f63f40073c0 tx=0x7f63f4009a50).stop 2021-03-09T08:52:00.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:00.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:00.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:00.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:00.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:52:00.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:00.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:00.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:00.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:00.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:00.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:00.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:00.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0046f30 con 0x7f63e00070a0 2021-03-09T08:52:00.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:00.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d120 con 0x7f6400145bd0 2021-03-09T08:52:00.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:00.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cf40 con 0x7f63e0064730 2021-03-09T08:52:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:00.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:00.878 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1258 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:00.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:00.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:00.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:00.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:52:00.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:00.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:00.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.865+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:52:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:52:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000fa40 con 0x7f63e0064730 2021-03-09T08:52:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:52:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:52:00.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:00.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:52:00.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.866+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4051df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a040 con 0x7f63e0064730 2021-03-09T08:52:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.867+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.867+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.867+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:00.893 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:00.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4015160 con 0x7f63e0064730 2021-03-09T08:52:00.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.867+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:00.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.867+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:52:00.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:00.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4006ca0 con 0x7f63e0064730 2021-03-09T08:52:01.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:01.365+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:01.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:01.365+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:02.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:02.365+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:02.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:02.365+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:02.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:02.365+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:03.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:03.365+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:03.365+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:04.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:04.365+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:04.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:04.365+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:04.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:04.380+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:05.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.365+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:05.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.365+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:05.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:52:05.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4051df0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4051df0).stop 2021-03-09T08:52:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:05.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:52:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:05.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:05.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030150 con 0x7f63e00070a0 2021-03-09T08:52:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:05.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:52:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f6400146050 2021-03-09T08:52:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:05.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:05.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:05.873 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1294 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:05.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:05.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.865+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.866+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:52:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.866+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.866+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.866+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.866+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.866+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:52:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:52:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e004deb0 con 0x7f6400146050 2021-03-09T08:52:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bfc0 con 0x7f6400146050 2021-03-09T08:52:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:52:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:52:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.867+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f63e0011450 tx=0x7f63f0006e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00066e0 con 0x7f6400146050 2021-03-09T08:52:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.869+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.869+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.869+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:05.882 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:05.883 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:05.883 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:05.883 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:05.884 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005b0f0 con 0x7f6400146050 2021-03-09T08:52:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.869+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.869+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:52:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:05.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T08:52:06.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:06.365+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:06.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:06.365+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:07.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:07.365+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:07.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:07.365+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:08.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:08.365+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:08.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:08.365+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:09.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:09.366+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:09.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:09.366+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:09.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:09.381+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:10.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.366+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:10.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.366+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:10.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:52:10.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:10.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f63e0011450 tx=0x7f63f0006e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:10.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7f63e0011450 tx=0x7f63f0006e00).stop 2021-03-09T08:52:10.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:10.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:10.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:52:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:10.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:10.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0046f30 con 0x7f63e00070a0 2021-03-09T08:52:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00235b0 con 0x7f6400145bd0 2021-03-09T08:52:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:10.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f63e0064730 2021-03-09T08:52:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:10.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.867+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.867+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:52:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.867+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:10.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.867+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:10.875 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:10.866+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1328 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.867+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.867+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.867+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:52:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:52:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:10.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:10.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bfc0 con 0x7f63e0064730 2021-03-09T08:52:10.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:10.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:10.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a8f0 con 0x7f63e0064730 2021-03-09T08:52:10.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:52:10.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.868+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:10.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.869+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:52:10.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.869+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:10.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.869+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f63fc05e010 tx=0x7f63fc047da0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:10.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f63e0064730 2021-03-09T08:52:10.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.871+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:10.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.871+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.871+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:10.885 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:10.886 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:10.886 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:10.889 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63e0064730 2021-03-09T08:52:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.874+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f63e0064730 2021-03-09T08:52:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:10.875+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:52:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:11.366+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:11.366+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:12.366+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:12.366+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:12.366+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:13.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:13.366+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:13.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:13.366+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:14.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:14.366+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:14.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:14.366+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:14.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:14.382+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:15.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.366+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:15.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.366+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:15.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:52:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f63fc05e010 tx=0x7f63fc047da0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7f63fc05e010 tx=0x7f63fc047da0).stop 2021-03-09T08:52:15.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:52:15.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:15.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:15.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:15.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:15.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030150 con 0x7f63e00070a0 2021-03-09T08:52:15.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:15.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:52:15.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:15.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cf40 con 0x7f6400146050 2021-03-09T08:52:15.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:15.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:15.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:15.874 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1365 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.867+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:15.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.868+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:52:15.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.868+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:15.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.868+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:15.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.868+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:15.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.868+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.868+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:52:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:52:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cb90 con 0x7f6400146050 2021-03-09T08:52:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000fa40 con 0x7f6400146050 2021-03-09T08:52:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:52:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.869+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.870+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021560 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f63f4047af0 tx=0x7f63f402bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.870+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:52:15.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.870+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:15.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f6400146050 2021-03-09T08:52:15.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.873+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:15.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.873+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:15.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.873+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:15.884 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:15.885 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:15.885 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:15.886 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:15.886 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:15.887 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:15.887 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:15.887 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:15.888 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:15.888 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40200e0 con 0x7f6400146050 2021-03-09T08:52:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.875+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.875+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:52:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:15.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:52:16.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:16.366+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:16.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:16.366+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:17.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:17.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:17.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:17.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:18.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:18.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:18.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:18.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:19.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:19.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:19.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:19.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:19.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:19.383+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:20.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:20.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:20.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:52:20.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:20.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021560 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f63f4047af0 tx=0x7f63f402bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:20.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021560 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f63f4047af0 tx=0x7f63f402bf40).stop 2021-03-09T08:52:20.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021560 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:52:20.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:20.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:20.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:20.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:20.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0046f30 con 0x7f63e00070a0 2021-03-09T08:52:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:20.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00229b0 con 0x7f6400145bd0 2021-03-09T08:52:20.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:20.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030150 con 0x7f63e0064730 2021-03-09T08:52:20.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:20.875 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1404 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:20.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:52:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:20.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.868+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:20.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:52:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:52:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d0d0 con 0x7f63e0064730 2021-03-09T08:52:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:52:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:52:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.869+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.870+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:52:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.870+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.870+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f63f003a7c0 tx=0x7f63f0054fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0015ec0 con 0x7f63e0064730 2021-03-09T08:52:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.871+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.871+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.871+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:20.887 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:20.887 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:20.887 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:20.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0015ec0 con 0x7f63e0064730 2021-03-09T08:52:20.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.873+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:20.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.873+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:52:20.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:20.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f63e0064730 2021-03-09T08:52:21.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:21.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:21.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:21.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:22.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:22.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:22.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:22.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:22.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:22.367+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:23.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:23.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:23.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:24.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:24.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:24.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:24.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:24.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:24.384+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:25.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:25.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:52:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f63f003a7c0 tx=0x7f63f0054fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7f63f003a7c0 tx=0x7f63f0054fb0).stop 2021-03-09T08:52:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b410 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:52:25.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:25.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:25.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:25.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:25.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:25.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:25.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:25.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:52:25.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:25.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f6400145bd0 2021-03-09T08:52:25.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:25.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0046f30 con 0x7f6400146050 2021-03-09T08:52:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.867+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.868+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.868+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.868+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:25.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:25.878 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:25.868+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1443 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:25.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.868+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:25.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.868+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:52:25.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.868+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.868+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:52:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:52:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e004deb0 con 0x7f6400146050 2021-03-09T08:52:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bfc0 con 0x7f6400146050 2021-03-09T08:52:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.870+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:52:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.870+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.870+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f550 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f63fc008e10 tx=0x7f63fc071280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.870+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:52:25.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.870+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:25.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06b0d0 con 0x7f6400146050 2021-03-09T08:52:25.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.872+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:25.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.872+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:25.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.872+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:25.887 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:25.888 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:25.888 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:25.888 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:25.888 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:25.888 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:25.889 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:25.889 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:25.889 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:25.889 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:25.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06b0d0 con 0x7f6400146050 2021-03-09T08:52:25.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.875+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:25.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.875+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:52:25.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:25.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f6400146050 2021-03-09T08:52:26.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:26.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:26.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:26.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:27.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:27.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:27.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:27.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:28.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:28.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:28.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:28.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:29.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:29.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:29.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:29.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:29.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:29.384+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:30.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.367+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:30.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.367+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:52:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f550 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f63fc008e10 tx=0x7f63fc071280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f550 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f63fc008e10 tx=0x7f63fc071280).stop 2021-03-09T08:52:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001f550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f550 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:52:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ccc0 con 0x7f63e00070a0 2021-03-09T08:52:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f6400145bd0 2021-03-09T08:52:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63e0064730 2021-03-09T08:52:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:30.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:30.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.869+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:30.878 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:30.868+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1489 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:52:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:30.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:30.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:30.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:30.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:30.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:52:30.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:52:30.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:30.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:30.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001ddb0 con 0x7f63e0064730 2021-03-09T08:52:30.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:30.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:30.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d0d0 con 0x7f63e0064730 2021-03-09T08:52:30.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:52:30.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.871+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:30.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.872+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f63f003a590 tx=0x7f63f002d550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:30.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.872+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:52:30.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.872+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:30.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002d110 con 0x7f63e0064730 2021-03-09T08:52:30.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.873+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:30.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.873+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:30.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.873+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:30.887 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:30.887 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:30.887 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:30.888 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:30.888 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:30.888 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:30.888 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:30.889 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:30.889 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:30.889 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:30.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002d110 con 0x7f63e0064730 2021-03-09T08:52:30.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.875+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:30.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.875+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:52:30.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:30.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a030 con 0x7f63e0064730 2021-03-09T08:52:31.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:31.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:31.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:31.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:32.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:32.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:32.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:32.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:32.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:32.368+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:33.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:33.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:33.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:34.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:34.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:34.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:34.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:34.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:34.385+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:35.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:35.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:35.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:52:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f63f003a590 tx=0x7f63f002d550).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f63f003a590 tx=0x7f63f002d550).stop 2021-03-09T08:52:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:52:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ca50 con 0x7f63e00070a0 2021-03-09T08:52:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400145bd0 2021-03-09T08:52:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fc30 con 0x7f6400146050 2021-03-09T08:52:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:52:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:35.886 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:35.869+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1515 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.870+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:52:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:52:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00151a0 con 0x7f6400146050 2021-03-09T08:52:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cb90 con 0x7f6400146050 2021-03-09T08:52:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:52:35.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.871+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:35.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.874+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:52:35.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.874+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:35.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.874+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000af40 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f63f405b3e0 tx=0x7f63f4005ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a040 con 0x7f6400146050 2021-03-09T08:52:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.875+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.875+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.875+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:35.896 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T08:52:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.877+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.877+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:52:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:35.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:52:36.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:36.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:36.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:36.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:37.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:37.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:37.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:37.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:38.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:38.368+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:38.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:38.368+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:39.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:39.369+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:39.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:39.369+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:39.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:39.386+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:40.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.369+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:40.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.369+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:40.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.869+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:52:40.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.869+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:40.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.869+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000af40 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f63f405b3e0 tx=0x7f63f4005ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:40.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.869+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000af40 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f63f405b3e0 tx=0x7f63f4005ed0).stop 2021-03-09T08:52:40.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:40.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:40.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:40.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000af40 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:40.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:52:40.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:40.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:40.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:40.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:52:40.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f6400145bd0 2021-03-09T08:52:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020670 con 0x7f63e0064730 2021-03-09T08:52:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:40.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:40.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:40.878 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1558 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:40.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:40.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:52:40.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:40.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:40.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:40.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:40.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:40.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:40.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.870+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:52:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:52:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d60 con 0x7f63e0064730 2021-03-09T08:52:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:52:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:52:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.871+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.872+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:52:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.872+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.872+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f63fc062290 tx=0x7f63fc01dee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0620f0 con 0x7f63e0064730 2021-03-09T08:52:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.873+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.873+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.873+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:40.889 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:40.889 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:40.889 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:40.890 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:40.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0620f0 con 0x7f63e0064730 2021-03-09T08:52:40.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.874+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:40.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.874+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:52:40.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:40.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f63e0064730 2021-03-09T08:52:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:41.369+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:41.369+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:42.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:42.369+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:42.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:42.369+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:42.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:42.369+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:43.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:43.369+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:43.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:43.369+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:44.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:44.369+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:44.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:44.369+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:44.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:44.387+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:45.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.369+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:45.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.369+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:52:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f63fc062290 tx=0x7f63fc01dee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:45.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f63fc062290 tx=0x7f63fc01dee0).stop 2021-03-09T08:52:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007b3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:45.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007b3f0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:52:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:45.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:45.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e00070a0 2021-03-09T08:52:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fc30 con 0x7f6400145bd0 2021-03-09T08:52:45.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:45.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00289c0 con 0x7f6400146050 2021-03-09T08:52:45.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:45.878 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1597 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:45.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.870+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:45.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.871+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:45.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.871+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:45.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.871+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:45.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:45.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.871+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:52:45.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.871+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:45.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.871+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:45.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:45.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:45.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:45.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:45.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:45.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:52:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.872+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:52:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.873+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:45.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.873+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:45.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0023790 con 0x7f6400146050 2021-03-09T08:52:45.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.873+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:45.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.873+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:45.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001ddb0 con 0x7f6400146050 2021-03-09T08:52:45.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.873+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:52:45.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.873+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:45.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c030 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f63f002d2b0 tx=0x7f63f000c150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:45.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.874+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:52:45.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.874+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:45.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002d2e0 con 0x7f6400146050 2021-03-09T08:52:45.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.876+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:45.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.876+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:45.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.876+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:45.888 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:45.888 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:45.890 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:45.890 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:45.890 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T08:52:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.880+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.880+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:52:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:45.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f6400146050 2021-03-09T08:52:46.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:46.369+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:46.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:46.369+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:47.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:47.370+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:47.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:47.370+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:48.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:48.370+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:48.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:48.370+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:49.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:49.370+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:49.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:49.370+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:49.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:49.388+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.370+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:50.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.370+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:50.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:52:50.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:50.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c030 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f63f002d2b0 tx=0x7f63f000c150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:50.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c030 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f63f002d2b0 tx=0x7f63f000c150).stop 2021-03-09T08:52:50.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:50.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:50.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:50.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c030 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:50.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:52:50.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:50.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:50.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0019f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:50.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:50.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:50.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:50.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:50.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004db30 con 0x7f63e00070a0 2021-03-09T08:52:50.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:50.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f3d0 con 0x7f6400145bd0 2021-03-09T08:52:50.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:50.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e0064730 2021-03-09T08:52:50.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:50.879 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1633 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:50.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:50.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:50.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:50.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.872+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:50.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.872+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0019f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:50.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.872+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:52:50.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.872+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:50.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.872+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:50.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.873+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:50.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.873+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:50.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.873+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:50.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.873+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:50.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:50.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:50.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:50.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:50.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:52:50.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:52:50.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:50.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:50.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bfc0 con 0x7f63e0064730 2021-03-09T08:52:50.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:50.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:50.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002d60 con 0x7f63e0064730 2021-03-09T08:52:50.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:52:50.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:50.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0019f40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc021fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:50.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:52:50.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.874+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:50.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc05a080 con 0x7f63e0064730 2021-03-09T08:52:50.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.875+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:50.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.875+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:50.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.875+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:50.890 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:50.890 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:50.890 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:50.891 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:50.891 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:50.891 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:50.891 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:50.892 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:50.892 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:50.892 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:50.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05a080 con 0x7f63e0064730 2021-03-09T08:52:50.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.878+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:50.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.878+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:52:50.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:50.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f63e0064730 2021-03-09T08:52:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:51.370+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:51.370+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:52.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:52.370+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:52.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:52.370+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:52.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:52.370+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:53.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:53.370+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:53.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:53.370+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:54.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:54.371+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:54.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:54.371+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:54.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:54.389+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:52:55.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.371+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:55.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.371+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:55.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0019f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:52:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0019f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:52:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0019f40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc021fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:52:55.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0019f40 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc021fb0).stop 2021-03-09T08:52:55.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:55.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:52:55.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0019f40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:52:55.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0019f40 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:52:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:52:55.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:55.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:52:55.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:55.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:52:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:52:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:52:55.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00146a0 con 0x7f63e00070a0 2021-03-09T08:52:55.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:55.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:52:55.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:52:55.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00669e0 con 0x7f6400146050 2021-03-09T08:52:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:52:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:52:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:52:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:52:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:52:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:52:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:52:55.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:52:55.887 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1673 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:52:55.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:52:55.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:52:55.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:52:55.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:52:55.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:55.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:55.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:52:55.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:52:55.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:52:55.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:52:55.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:52:55.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:55.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T08:52:55.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:52:55.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:52:55.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0023790 con 0x7f6400146050 2021-03-09T08:52:55.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:52:55.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:55.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007b3f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f63e0014080 tx=0x7f63f005cef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:52:55.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:52:55.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.873+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:52:55.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00182e0 con 0x7f6400146050 2021-03-09T08:52:55.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.874+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:52:55.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.874+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:52:55.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.874+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:52:55.894 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:52:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:52:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:52:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:52:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:52:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:52:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:52:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:52:55.897 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:52:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00153e0 con 0x7f6400146050 2021-03-09T08:52:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.874+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:52:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.874+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:52:55.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:55.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T08:52:56.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:56.371+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:56.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:56.371+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:57.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:57.371+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:57.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:57.371+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:58.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:58.371+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:58.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:58.371+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:59.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:59.371+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:52:59.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:59.371+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:52:59.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:52:59.390+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:00.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.371+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:00.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.371+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:53:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:00.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007b3f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f63e0014080 tx=0x7f63f005cef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:00.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007b3f0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f63e0014080 tx=0x7f63f005cef0).stop 2021-03-09T08:53:00.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:00.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:00.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007b3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:00.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007b3f0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:00.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:53:00.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:00.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:00.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:00.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:00.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00162b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:00.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:00.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:00.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020670 con 0x7f63e00070a0 2021-03-09T08:53:00.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:00.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400145bd0 2021-03-09T08:53:00.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:00.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:53:00.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:00.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:00.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:00.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:00.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:00.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:00.883 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1709 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:53:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:53:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:53:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:53:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:00.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T08:53:00.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:53:00.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.874+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:00.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.875+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:53:00.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.875+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00162b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f002fee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:00.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.875+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00534e0 con 0x7f63e0064730 2021-03-09T08:53:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.877+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.877+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.877+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:00.892 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0064730 2021-03-09T08:53:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.877+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.877+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:53:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:00.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:53:01.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:01.371+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:01.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:01.371+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:02.372+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:02.372+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:02.372+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:03.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:03.372+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:03.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:03.372+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:04.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:04.372+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:04.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:04.372+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:04.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:04.391+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:05.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.372+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:05.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.372+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:53:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:05.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00162b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f002fee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00162b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f002fee0).stop 2021-03-09T08:53:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00162b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:05.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00162b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:53:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:05.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:05.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028770 con 0x7f63e00070a0 2021-03-09T08:53:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00669e0 con 0x7f6400145bd0 2021-03-09T08:53:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00146a0 con 0x7f6400146050 2021-03-09T08:53:05.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.873+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.874+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.874+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:05.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:05.882 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:05.874+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1743 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.874+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.874+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:53:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.874+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:05.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.874+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.875+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.875+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.875+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.875+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:05.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.875+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.875+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:53:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:53:05.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:05.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:05.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0046f30 con 0x7f6400146050 2021-03-09T08:53:05.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00151a0 con 0x7f6400146050 2021-03-09T08:53:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:53:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:53:05.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:05.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.876+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019f40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4044400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:05.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f6400146050 2021-03-09T08:53:05.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.877+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:05.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.877+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:05.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.877+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:05.890 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:05.890 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:05.892 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400b470 con 0x7f6400146050 2021-03-09T08:53:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.879+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.879+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:53:05.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:05.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:53:06.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:06.372+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:06.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:06.372+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:07.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:07.372+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:07.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:07.372+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:08.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:08.372+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:08.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:08.372+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:09.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:09.372+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:09.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:09.372+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:09.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:09.392+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:10.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.372+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:10.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.372+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:10.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:53:10.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019f40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4044400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019f40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4044400).stop 2021-03-09T08:53:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:10.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019f40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019f40 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:53:10.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:10.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:10.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:10.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:53:10.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:10.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ebc0 con 0x7f6400145bd0 2021-03-09T08:53:10.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:10.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028770 con 0x7f63e0064730 2021-03-09T08:53:10.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:10.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:10.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:10.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:10.884 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:10.874+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1779 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:10.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.875+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:10.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.875+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:53:10.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.875+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:10.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.875+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:10.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.875+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:10.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.875+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.875+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:53:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:53:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T08:53:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:53:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:53:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.876+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.877+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:53:10.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.877+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006dc00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f00463f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:10.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.877+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:10.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00460b0 con 0x7f63e0064730 2021-03-09T08:53:10.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.878+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:10.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.878+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:10.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.878+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:10.893 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:10.894 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:10.894 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:10.894 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:10.894 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:10.895 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:10.895 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:10.895 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:10.896 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00460b0 con 0x7f63e0064730 2021-03-09T08:53:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.881+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.881+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:53:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:10.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:53:11.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:11.373+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:11.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:11.373+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:12.373+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:12.373+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:12.373+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:13.373+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:13.373+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:14.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:14.373+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:14.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:14.373+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:14.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:14.393+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:15.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.373+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:15.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.373+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:53:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006dc00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f00463f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:15.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006dc00 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f00463f0).stop 2021-03-09T08:53:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e006dc00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:15.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006dc00 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:53:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:15.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00187b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b3f0 con 0x7f63e00070a0 2021-03-09T08:53:15.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:15.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400145bd0 2021-03-09T08:53:15.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:15.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013cf0 con 0x7f6400146050 2021-03-09T08:53:15.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:15.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:15.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:15.883 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1815 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:15.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:15.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.875+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:15.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.876+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00187b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:15.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.876+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:53:15.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.876+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:15.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.876+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:15.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.877+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:15.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.877+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.877+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0069140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:53:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:53:15.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:15.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:15.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d60 con 0x7f6400146050 2021-03-09T08:53:15.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T08:53:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:53:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.878+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00187b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f63e002cd70 tx=0x7f63f00211f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.879+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:53:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.879+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d320 con 0x7f6400146050 2021-03-09T08:53:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.880+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.880+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.880+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:15.896 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:15.896 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:15.896 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:15.896 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:15.896 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:15.897 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d320 con 0x7f6400146050 2021-03-09T08:53:15.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.880+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:15.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.880+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:53:15.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:15.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f6400146050 2021-03-09T08:53:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:16.373+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:16.373+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:17.374+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:17.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:17.374+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:18.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:18.374+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:18.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:18.374+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:19.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:19.374+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:19.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:19.374+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:19.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:19.394+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:20.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.374+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:20.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.374+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:20.878 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1858 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00187b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:53:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00187b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:20.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00187b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f63e002cd70 tx=0x7f63f00211f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00187b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f63e002cd70 tx=0x7f63f00211f0).stop 2021-03-09T08:53:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00187b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:20.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00187b0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:53:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:20.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:20.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022bb0 con 0x7f63e00070a0 2021-03-09T08:53:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:53:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028770 con 0x7f63e0064730 2021-03-09T08:53:20.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:20.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.876+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.877+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.877+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:53:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.877+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:20.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.877+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:20.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.878+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:20.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.878+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:20.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.878+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:20.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:20.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:20.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:20.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:53:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:53:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:20.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:53:20.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:20.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:20.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002d60 con 0x7f63e0064730 2021-03-09T08:53:20.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:53:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:53:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000cf40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f63f002e500 tx=0x7f63f000efe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006d90 con 0x7f63e0064730 2021-03-09T08:53:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.882+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.882+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.882+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:20.895 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002e340 con 0x7f63e0064730 2021-03-09T08:53:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.882+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.882+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:53:20.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:20.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T08:53:21.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:21.374+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:21.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:21.374+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:22.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:22.374+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:22.374+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:22.374+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:23.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:23.374+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:23.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:23.374+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:24.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:24.375+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:24.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:24.375+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:24.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:24.395+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:25.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.374+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:25.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.374+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:53:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000cf40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f63f002e500 tx=0x7f63f000efe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000cf40 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f63f002e500 tx=0x7f63f000efe0).stop 2021-03-09T08:53:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:25.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000cf40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000cf40 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:53:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:25.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:25.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:25.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:25.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00239e0 con 0x7f63e00070a0 2021-03-09T08:53:25.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:25.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013cf0 con 0x7f6400145bd0 2021-03-09T08:53:25.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:25.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:53:25.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:25.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:25.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:25.894 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:25.877+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1893 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:25.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:25.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:25.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:53:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.878+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:25.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:25.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:53:25.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:53:25.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:25.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0023790 con 0x7f6400146050 2021-03-09T08:53:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:25.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bfc0 con 0x7f6400146050 2021-03-09T08:53:25.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:53:25.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.879+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:25.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.880+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:53:25.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.880+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc072ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:25.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.880+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:25.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T08:53:25.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.881+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:25.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.881+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:25.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.881+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:25.917 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:25.919 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T08:53:25.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.882+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:25.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.882+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:53:25.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:25.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f6400146050 2021-03-09T08:53:26.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:26.375+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:26.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:26.375+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:27.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:27.375+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:27.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:27.375+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:28.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:28.375+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:28.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:28.375+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:29.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:29.375+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:29.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:29.375+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:29.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:29.396+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:30.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.375+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:30.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.375+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.877+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:53:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.877+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.877+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc072ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.877+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc072ea0).stop 2021-03-09T08:53:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:30.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:30.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:30.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:53:30.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:30.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:30.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00174e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:30.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:30.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:30.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:30.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:30.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027f90 con 0x7f63e00070a0 2021-03-09T08:53:30.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:30.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020f40 con 0x7f6400145bd0 2021-03-09T08:53:30.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:30.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00239e0 con 0x7f63e0064730 2021-03-09T08:53:30.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:30.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:30.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:30.886 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1927 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:30.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:30.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.878+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:30.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00174e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:30.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.879+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:53:30.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.879+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:30.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.879+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:30.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:30.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:30.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:30.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:30.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:30.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:30.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:30.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:30.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:53:30.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:53:30.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:30.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:30.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T08:53:30.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:30.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:30.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.880+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0023790 con 0x7f63e0064730 2021-03-09T08:53:30.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.881+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:53:30.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.881+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:30.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.881+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00174e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f63e00131c0 tx=0x7f63fc01a330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:30.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.881+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:53:30.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.881+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:30.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047cc0 con 0x7f63e0064730 2021-03-09T08:53:30.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.881+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:30.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.881+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:30.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.881+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:30.901 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:30.902 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:30.902 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:30.902 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:30.902 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:30.903 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:30.903 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:30.903 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:30.903 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:30.903 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:30.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047cc0 con 0x7f63e0064730 2021-03-09T08:53:30.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.884+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:30.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.884+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:53:30.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:30.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00abe0 con 0x7f63e0064730 2021-03-09T08:53:31.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:31.375+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:31.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:31.375+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:32.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:32.376+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:32.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:32.376+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:32.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:32.376+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:33.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:33.376+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:33.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:33.376+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:34.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:34.375+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:34.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:34.375+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:34.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:34.397+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:35.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.375+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:35.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.376+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:35.881 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 1965 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00174e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:53:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00174e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00174e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f63e00131c0 tx=0x7f63fc01a330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00174e0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f63e00131c0 tx=0x7f63fc01a330).stop 2021-03-09T08:53:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00174e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00174e0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:53:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:35.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:35.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:35.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001de00 con 0x7f63e00070a0 2021-03-09T08:53:35.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004fcf0 con 0x7f6400145bd0 2021-03-09T08:53:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a120 con 0x7f6400146050 2021-03-09T08:53:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.879+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.880+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.880+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:53:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.880+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.880+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.880+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.880+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.880+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:53:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:53:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f380 con 0x7f6400146050 2021-03-09T08:53:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T08:53:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:53:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:35.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:53:35.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc005f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:35.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.882+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:35.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc022120 con 0x7f6400146050 2021-03-09T08:53:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.884+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.884+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.884+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:35.900 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:35.902 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc022120 con 0x7f6400146050 2021-03-09T08:53:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.885+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.885+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:53:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:35.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc05b880 con 0x7f6400146050 2021-03-09T08:53:36.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:36.376+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:36.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:36.376+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:37.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:37.376+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:37.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:37.376+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:38.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:38.376+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:38.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:38.376+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:39.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:39.377+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:39.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:39.377+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:39.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:39.398+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:40.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.377+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:40.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.377+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:53:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc005f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc005f00).stop 2021-03-09T08:53:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c620 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:53:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00663f0 con 0x7f63e00070a0 2021-03-09T08:53:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:53:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027f90 con 0x7f63e0064730 2021-03-09T08:53:40.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.879+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:40.889 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2006 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:40.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:40.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:40.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:40.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:53:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.880+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:53:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:53:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001c870 con 0x7f63e0064730 2021-03-09T08:53:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e004deb0 con 0x7f63e0064730 2021-03-09T08:53:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:53:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:53:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f63f000b4d0 tx=0x7f63f00462c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.881+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:40.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d320 con 0x7f63e0064730 2021-03-09T08:53:40.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.882+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:40.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.882+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:40.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.882+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:40.898 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:40.900 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000b0e0 con 0x7f63e0064730 2021-03-09T08:53:40.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.882+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:40.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.882+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:53:40.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:40.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:53:41.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:41.377+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:41.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:41.377+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:42.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:42.377+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:42.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:42.377+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:42.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:42.377+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:43.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:43.377+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:43.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:43.377+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:44.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:44.377+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:44.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:44.377+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:44.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:44.399+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:45.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.377+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:45.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.377+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:45.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:53:45.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:45.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f63f000b4d0 tx=0x7f63f00462c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:45.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 rx=0x7f63f000b4d0 tx=0x7f63f00462c0).stop 2021-03-09T08:53:45.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:45.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000ee10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:53:45.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:45.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:45.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:45.885 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2042 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:45.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:45.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:45.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020670 con 0x7f63e00070a0 2021-03-09T08:53:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400145bd0 2021-03-09T08:53:45.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014370 con 0x7f6400146050 2021-03-09T08:53:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.880+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:45.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.881+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:45.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.881+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:53:45.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.881+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:45.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.881+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:45.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.881+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:45.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.881+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:45.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.881+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:45.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:45.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:53:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:53:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025480 con 0x7f6400146050 2021-03-09T08:53:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0046f30 con 0x7f6400146050 2021-03-09T08:53:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:53:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.882+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.883+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cca0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f63f4015320 tx=0x7f63f4009830).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.883+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:53:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.883+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f6400146050 2021-03-09T08:53:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.884+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.884+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.884+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:45.901 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:45.901 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:45.901 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:45.901 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:45.902 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f6400146050 2021-03-09T08:53:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.884+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.884+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:53:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:45.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:53:46.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:46.378+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:46.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:46.378+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:47.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:47.378+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:47.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:47.378+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:48.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:48.378+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:49.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:48.378+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:49.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:49.378+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:49.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:49.378+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:49.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:49.400+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:50.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.378+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:50.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.378+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:50.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:53:50.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:50.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cca0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f63f4015320 tx=0x7f63f4009830).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:50.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cca0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f63f4015320 tx=0x7f63f4009830).stop 2021-03-09T08:53:50.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:50.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:50.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007cca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:50.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cca0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:50.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:53:50.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:50.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:50.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:50.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:50.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:50.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:50.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:50.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a120 con 0x7f63e00070a0 2021-03-09T08:53:50.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:50.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f5d0 con 0x7f6400145bd0 2021-03-09T08:53:50.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:50.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:53:50.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:50.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:50.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:50.889 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2082 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:50.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:50.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:50.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:50.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:53:50.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:50.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.881+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:50.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:50.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:50.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:50.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:50.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:50.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:50.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:50.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:50.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:53:50.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:53:50.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:50.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:50.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d540 con 0x7f63e0064730 2021-03-09T08:53:50.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:50.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:50.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f380 con 0x7f63e0064730 2021-03-09T08:53:50.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:53:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f63fc06d040 tx=0x7f63fc075fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:53:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.882+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T08:53:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.885+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.885+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.885+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:50.899 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc021270 con 0x7f63e0064730 2021-03-09T08:53:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.885+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.885+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:53:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:50.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63e0064730 2021-03-09T08:53:51.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:51.378+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:51.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:51.378+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:52.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:52.378+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:52.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:52.378+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:52.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:52.378+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:53.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:53.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:53.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:53.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:54.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:54.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:54.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:54.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:54.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:54.401+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:53:55.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:55.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:55.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:53:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:53:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f63fc06d040 tx=0x7f63fc075fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:53:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f63fc06d040 tx=0x7f63fc075fe0).stop 2021-03-09T08:53:55.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:53:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000ee10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:53:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ee10 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:53:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:53:55.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:55.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:53:55.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:55.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:53:55.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:53:55.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:53:55.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:55.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018100 con 0x7f63e00070a0 2021-03-09T08:53:55.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:55.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077b70 con 0x7f6400145bd0 2021-03-09T08:53:55.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:53:55.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027f90 con 0x7f6400146050 2021-03-09T08:53:55.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:53:55.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:53:55.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:55.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:53:55.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:53:55.890 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:53:55.882+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2118 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:53:55.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.883+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:53:55.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.883+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:53:55.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.883+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:53:55.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.883+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:53:55.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.883+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:53:55.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.883+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:53:55.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.883+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:53:55.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:53:55.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:55.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:55.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:53:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:53:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:53:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:53:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:53:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f380 con 0x7f6400146050 2021-03-09T08:53:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:53:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:53:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025480 con 0x7f6400146050 2021-03-09T08:53:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:53:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:53:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:53:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f4033420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:53:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f6400146050 2021-03-09T08:53:55.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.887+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:53:55.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.887+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:53:55.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.887+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:53:55.898 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:53:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:53:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:53:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:53:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:53:55.900 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:53:55.900 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:53:55.900 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:53:55.900 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:53:55.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400b540 con 0x7f6400146050 2021-03-09T08:53:55.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.887+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:53:55.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.887+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:53:55.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:55.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:53:56.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:56.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:56.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:56.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:57.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:57.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:57.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:57.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:58.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:58.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:58.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:58.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:59.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:59.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:53:59.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:59.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:53:59.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:53:59.402+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:54:00.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f4033420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f4033420).stop 2021-03-09T08:54:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:00.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:00.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:54:00.887 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2152 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:00.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:00.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:54:00.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079fa0 con 0x7f6400145bd0 2021-03-09T08:54:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cc80 con 0x7f63e0064730 2021-03-09T08:54:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:00.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.883+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:00.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:54:00.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:00.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:00.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:00.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:00.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:00.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:00.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:00.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:00.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:00.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:00.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:54:00.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:54:00.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:00.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:00.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a740 con 0x7f63e0064730 2021-03-09T08:54:00.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:00.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:00.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001c870 con 0x7f63e0064730 2021-03-09T08:54:00.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:54:00.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.884+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:00.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d850 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f63f0006b00 tx=0x7f63f0046050).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:00.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.885+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:54:00.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.885+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:00.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f63e0064730 2021-03-09T08:54:00.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.886+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:00.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.886+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:00.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.886+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:00.900 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:00.900 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:00.900 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:00.900 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:00.901 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:00.901 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:00.901 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:00.901 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:00.901 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:00.902 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:00.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d7a0 con 0x7f63e0064730 2021-03-09T08:54:00.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.886+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:00.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.886+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:54:00.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:00.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:54:01.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:01.380+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:01.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:01.380+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:02.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:02.380+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:02.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:02.380+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:02.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:02.380+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:03.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:03.380+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:03.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:03.380+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:04.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:04.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:04.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:04.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:04.403+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:05.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.379+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:05.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.379+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:05.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:54:05.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:05.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d850 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f63f0006b00 tx=0x7f63f0046050).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:05.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d850 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 rx=0x7f63f0006b00 tx=0x7f63f0046050).stop 2021-03-09T08:54:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d850 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:05.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:54:05.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:05.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:05.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:05.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:05.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:05.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:05.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:05.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e00070a0 2021-03-09T08:54:05.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00210f0 con 0x7f6400145bd0 2021-03-09T08:54:05.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027f90 con 0x7f6400146050 2021-03-09T08:54:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:05.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:54:05.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:05.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:05.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:05.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:05.895 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2192 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:05.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:05.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:05.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:05.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:05.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:05.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:05.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:05.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:05.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:54:05.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:54:05.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:05.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:05.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e000 con 0x7f6400146050 2021-03-09T08:54:05.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:05.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:05.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f380 con 0x7f6400146050 2021-03-09T08:54:05.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:54:05.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.884+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:05.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.885+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f63f4029570 tx=0x7f63f4029730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:05.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.885+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:54:05.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.885+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:05.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c020 con 0x7f6400146050 2021-03-09T08:54:05.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.886+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:05.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.886+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:05.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.886+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:05.902 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:05.903 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:05.903 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:05.903 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:05.903 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:05.904 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:05.904 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:05.904 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:05.904 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:05.905 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:05.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4047740 con 0x7f6400146050 2021-03-09T08:54:05.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.886+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:05.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.886+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:54:05.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:05.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:54:06.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:06.380+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:06.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:06.380+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:07.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:07.380+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:07.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:07.380+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:08.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:08.381+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:08.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:08.381+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:09.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:09.380+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:09.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:09.380+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:09.404+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:10.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.381+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:10.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.381+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:10.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:54:10.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:10.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f63f4029570 tx=0x7f63f4029730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 rx=0x7f63f4029570 tx=0x7f63f4029730).stop 2021-03-09T08:54:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:10.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c600 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:54:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:10.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0012300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:10.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00256d0 con 0x7f63e00070a0 2021-03-09T08:54:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002dcc0 con 0x7f6400145bd0 2021-03-09T08:54:10.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:10.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:54:10.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:10.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:10.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:10.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:10.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:10.893 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:10.885+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2239 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:10.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.885+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0012300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:10.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.885+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:54:10.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.885+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:10.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.885+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.885+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.885+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.885+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:10.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:54:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:54:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bdd0 con 0x7f63e0064730 2021-03-09T08:54:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d540 con 0x7f63e0064730 2021-03-09T08:54:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:54:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0012300 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:54:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.887+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T08:54:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.889+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:10.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.889+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:10.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.889+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:10.901 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:10.901 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:10.903 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e0064730 2021-03-09T08:54:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.889+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.889+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:54:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:10.890+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T08:54:11.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:11.380+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:11.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:11.380+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:12.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:12.381+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:12.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:12.381+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:12.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:12.381+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:13.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:13.381+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:13.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:13.381+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:14.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:14.381+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:14.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:14.381+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:14.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:14.405+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:15.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.382+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:15.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.382+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:15.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0012300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:54:15.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0012300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:15.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0012300 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:15.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0012300 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009a70).stop 2021-03-09T08:54:15.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0012300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0012300 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:54:15.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:15.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:15.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018140 con 0x7f63e00070a0 2021-03-09T08:54:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066170 con 0x7f6400145bd0 2021-03-09T08:54:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:15.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a050 con 0x7f6400146050 2021-03-09T08:54:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.885+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:15.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:15.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:15.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:54:15.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:15.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:15.898 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2263 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:15.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:15.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:15.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:15.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:15.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:15.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:54:15.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:54:15.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00253f0 con 0x7f6400146050 2021-03-09T08:54:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:15.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.886+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e000 con 0x7f6400146050 2021-03-09T08:54:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.887+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:54:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.887+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.887+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:54:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.887+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:15.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.887+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c5e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f63f400d6d0 tx=0x7f63f4051650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:15.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a040 con 0x7f6400146050 2021-03-09T08:54:15.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.889+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:15.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.889+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:15.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.889+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:15.906 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:15.906 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:15.906 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:15.906 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:15.907 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:15.907 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:15.907 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:15.907 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:15.907 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:15.908 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:15.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40098a0 con 0x7f6400146050 2021-03-09T08:54:15.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.889+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:15.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.889+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:54:15.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:15.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T08:54:16.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:16.381+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:16.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:16.381+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:17.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:17.382+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:17.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:17.382+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:18.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:18.382+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:18.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:18.382+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:19.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:19.382+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:19.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:19.382+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:19.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:19.406+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:20.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.381+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:20.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.381+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:20.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:54:20.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c5e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f63f400d6d0 tx=0x7f63f4051650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c5e0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7f63f400d6d0 tx=0x7f63f4051650).stop 2021-03-09T08:54:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:20.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:20.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c5e0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:20.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:54:20.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:20.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:20.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020f90 con 0x7f63e00070a0 2021-03-09T08:54:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f0e0 con 0x7f6400145bd0 2021-03-09T08:54:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:20.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:54:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:20.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:54:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:20.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:20.896 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2308 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:20.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:20.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:54:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:54:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00112f0 con 0x7f63e0064730 2021-03-09T08:54:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bdd0 con 0x7f63e0064730 2021-03-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.887+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.888+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc076130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:20.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.888+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:54:20.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.888+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:20.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.889+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.889+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.889+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:20.904 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:20.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T08:54:20.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.889+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:20.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.889+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:54:20.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:20.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc05b170 con 0x7f63e0064730 2021-03-09T08:54:21.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:21.382+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:21.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:21.382+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:22.382+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:22.382+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:22.382+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:23.383+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:23.383+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:24.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:24.383+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:24.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:24.383+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:24.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:24.407+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:25.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.382+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:25.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.382+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:25.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:54:25.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:25.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc076130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:25.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5e0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc076130).stop 2021-03-09T08:54:25.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:25.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:25.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:25.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5e0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:25.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:54:25.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:25.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:25.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:25.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:25.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:25.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:25.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:25.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027db0 con 0x7f63e00070a0 2021-03-09T08:54:25.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:25.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bbc0 con 0x7f6400145bd0 2021-03-09T08:54:25.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:25.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016100 con 0x7f6400146050 2021-03-09T08:54:25.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:25.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:25.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:25.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:25.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:25.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:54:25.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:25.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:25.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:25.897 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:25.888+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2338 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:25.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:25.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:25.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:25.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:25.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:54:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:54:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013f20 con 0x7f6400146050 2021-03-09T08:54:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a740 con 0x7f6400146050 2021-03-09T08:54:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:54:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.889+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.890+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f005d340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.890+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:54:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.890+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.890+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T08:54:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.890+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.890+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.890+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:25.905 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:25.905 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:25.906 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:25.906 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:25.907 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:25.907 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003b480 con 0x7f6400146050 2021-03-09T08:54:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.891+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.891+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:54:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:25.893+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f6400146050 2021-03-09T08:54:26.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:26.382+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:26.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:26.382+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:27.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:27.382+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:27.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:27.382+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:28.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:28.383+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:28.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:28.383+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:29.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:29.383+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:29.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:29.383+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:29.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:29.408+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:30.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.384+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:30.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.384+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:30.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:54:30.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:30.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f005d340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:30.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f005d340).stop 2021-03-09T08:54:30.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:30.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:30.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:30.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:30.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:54:30.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:30.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:30.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:30.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:30.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:30.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:30.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:30.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077880 con 0x7f63e00070a0 2021-03-09T08:54:30.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:30.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:54:30.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:30.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f430 con 0x7f63e0064730 2021-03-09T08:54:30.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:30.902 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2384 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:30.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:30.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:30.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:30.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:30.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:30.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:54:30.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:30.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.889+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:30.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:30.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:30.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:30.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:30.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:30.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:30.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:30.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:30.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:54:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:54:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T08:54:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013f20 con 0x7f63e0064730 2021-03-09T08:54:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:54:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.890+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.891+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001bac0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f0048440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:30.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.891+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:54:30.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.891+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:30.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.893+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003e810 con 0x7f63e0064730 2021-03-09T08:54:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.893+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.893+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.893+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:30.924 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:30.924 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:30.926 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d320 con 0x7f63e0064730 2021-03-09T08:54:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.897+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.897+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:54:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:30.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:54:31.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:31.384+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:31.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:31.384+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:32.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:32.383+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:32.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:32.383+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:32.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:32.383+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:33.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:33.384+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:33.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:33.384+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:34.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:34.384+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:34.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:34.384+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:34.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:34.409+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:35.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.384+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:35.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.384+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:54:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001bac0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f0048440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001bac0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f0048440).stop 2021-03-09T08:54:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:35.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:35.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001bac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:35.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001bac0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:35.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:54:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:35.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:35.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d590 con 0x7f63e00070a0 2021-03-09T08:54:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b9c0 con 0x7f6400145bd0 2021-03-09T08:54:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026fc0 con 0x7f6400146050 2021-03-09T08:54:35.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:35.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:54:35.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:35.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:35.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:35.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:35.899 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:35.889+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2424 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:54:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:54:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029250 con 0x7f6400146050 2021-03-09T08:54:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00112f0 con 0x7f6400146050 2021-03-09T08:54:35.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:54:35.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.890+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:35.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.891+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f63fc047a90 tx=0x7f63fc05b420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:35.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.891+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:54:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.891+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.893+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047cc0 con 0x7f6400146050 2021-03-09T08:54:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.893+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.893+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.893+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:35.907 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:35.907 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:35.907 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:35.907 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:35.908 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:35.908 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:35.908 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:35.908 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:35.908 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:35.909 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:35.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.893+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T08:54:35.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.893+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:35.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.893+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:54:35.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:35.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f6400146050 2021-03-09T08:54:36.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:36.384+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:36.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:36.384+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:37.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:37.385+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:37.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:37.385+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:38.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:38.385+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:38.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:38.385+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:39.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:39.385+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:39.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:39.385+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:39.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:39.410+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:40.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.385+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:40.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.385+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:54:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f63fc047a90 tx=0x7f63fc05b420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7f63fc047a90 tx=0x7f63fc05b420).stop 2021-03-09T08:54:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:54:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012e80 con 0x7f63e00070a0 2021-03-09T08:54:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016100 con 0x7f6400145bd0 2021-03-09T08:54:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bbc0 con 0x7f63e0064730 2021-03-09T08:54:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:40.898 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2457 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.890+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.891+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.892+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:54:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.892+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.892+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:40.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.892+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:40.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.892+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:40.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.892+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:40.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:40.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:54:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:54:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016e20 con 0x7f63e0064730 2021-03-09T08:54:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00253f0 con 0x7f63e0064730 2021-03-09T08:54:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:54:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.893+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4044c90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.894+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:54:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.894+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f63e0064730 2021-03-09T08:54:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.895+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.895+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:40.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.895+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:40.906 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:40.906 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:40.906 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:40.908 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:40.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63e0064730 2021-03-09T08:54:40.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.897+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:40.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.897+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:54:40.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:40.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T08:54:41.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:41.385+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:41.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:41.385+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:42.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:42.385+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:42.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:42.385+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:42.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:42.385+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:43.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:43.385+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:43.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:43.385+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:44.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:44.386+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:44.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:44.386+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:44.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:44.411+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:45.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.386+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:45.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.386+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:45.893 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2488 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:45.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:54:45.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:45.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4044c90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4044c90).stop 2021-03-09T08:54:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001f3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:54:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:54:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027db0 con 0x7f6400145bd0 2021-03-09T08:54:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f6400146050 2021-03-09T08:54:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.891+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.892+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.892+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.892+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.892+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:54:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.892+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.892+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.893+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.893+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.893+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:45.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:54:45.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:54:45.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:45.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065e20 con 0x7f6400146050 2021-03-09T08:54:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016e20 con 0x7f6400146050 2021-03-09T08:54:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:54:45.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:45.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f400bd10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:45.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.895+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:54:45.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.895+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:45.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c040 con 0x7f6400146050 2021-03-09T08:54:45.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.896+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:45.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.896+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:45.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.896+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:45.916 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:45.916 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:45.918 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f6400146050 2021-03-09T08:54:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.898+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.898+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:54:45.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:45.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:54:46.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:46.386+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:46.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:46.386+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:47.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:47.386+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:47.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:47.386+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:48.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:48.386+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:48.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:48.386+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:49.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:49.386+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:49.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:49.386+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:49.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:49.412+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:50.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.386+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:50.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.386+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:50.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:54:50.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:50.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f400bd10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:50.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f400bd10).stop 2021-03-09T08:54:50.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:50.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:54:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e00070a0 2021-03-09T08:54:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a1a0 con 0x7f6400145bd0 2021-03-09T08:54:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d590 con 0x7f63e0064730 2021-03-09T08:54:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:50.901 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:50.892+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2532 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:50.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.893+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.893+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:54:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.893+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.893+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:54:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:54:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b9c0 con 0x7f63e0064730 2021-03-09T08:54:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.895+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.895+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.895+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065e20 con 0x7f63e0064730 2021-03-09T08:54:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.895+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:54:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.895+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.895+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4009350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.895+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:54:50.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.895+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:50.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f63e0064730 2021-03-09T08:54:50.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.897+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:50.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.897+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:50.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.897+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:50.911 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:50.911 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:50.912 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:50.912 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:50.912 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:50.912 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:50.913 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:50.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63e0064730 2021-03-09T08:54:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.900+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.900+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:54:50.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:50.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f63e0064730 2021-03-09T08:54:51.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:51.386+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:51.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:51.386+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:52.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:52.387+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:52.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:52.387+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:52.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:52.387+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:53.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:53.387+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:53.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:53.387+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:54.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:54.387+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:54.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:54.387+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:54.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:54.413+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:54:55.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.387+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:55.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.387+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:54:55.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:54:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4009350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:54:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4009350).stop 2021-03-09T08:54:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:55.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:54:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001f3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:54:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001f3d0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:54:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:54:55.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:55.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:54:55.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:55.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:54:55.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:54:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:54:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:54:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:55.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f6400145bd0 2021-03-09T08:54:55.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:54:55.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018340 con 0x7f6400146050 2021-03-09T08:54:55.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:54:55.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:54:55.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:55.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:54:55.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:54:55.902 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2566 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:54:55.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:54:55.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:54:55.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:54:55.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.893+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:54:55.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:54:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:54:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:54:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:54:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:55.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:55.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:54:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:54:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:54:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:54:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:54:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00102f0 con 0x7f6400146050 2021-03-09T08:54:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:54:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:54:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b9c0 con 0x7f6400146050 2021-03-09T08:54:55.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:54:55.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.894+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:55.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.895+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:54:55.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.895+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:54:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.895+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f63f402a590 tx=0x7f63f4029180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:54:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c040 con 0x7f6400146050 2021-03-09T08:54:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.896+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:54:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.896+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:54:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.896+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:54:55.912 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:54:55.912 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:54:55.912 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:55.912 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:54:55.913 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:54:55.913 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:54:55.913 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:54:55.914 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:54:55.914 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:54:55.914 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:54:55.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404b270 con 0x7f6400146050 2021-03-09T08:54:55.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.896+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:54:55.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.896+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:54:55.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:55.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402aac0 con 0x7f6400146050 2021-03-09T08:54:56.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:56.387+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:56.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:56.387+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:57.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:57.387+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:57.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:57.387+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:58.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:58.387+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:58.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:58.387+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:59.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:59.387+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:54:59.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:59.387+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:54:59.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:54:59.414+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:00.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.388+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:00.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.388+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:00.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:00.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:00.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f63f402a590 tx=0x7f63f4029180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:00.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f63f402a590 tx=0x7f63f4029180).stop 2021-03-09T08:55:00.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:00.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:00.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c550 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:00.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c550 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:00.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:55:00.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:00.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:00.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:00.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:00.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:00.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:00.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:00.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:55:00.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:00.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013ad0 con 0x7f6400145bd0 2021-03-09T08:55:00.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:00.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018340 con 0x7f63e0064730 2021-03-09T08:55:00.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:00.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:00.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:00.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:00.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:00.903 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:00.894+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2602 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:00.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:00.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:55:00.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:00.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:00.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:00.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:00.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:00.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:55:00.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:55:00.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:00.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:00.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.895+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f3e0 con 0x7f63e0064730 2021-03-09T08:55:00.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.896+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:00.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.896+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:00.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.896+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00102f0 con 0x7f63e0064730 2021-03-09T08:55:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.896+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:55:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.896+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.896+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:55:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.896+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ffc0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f63f4005f10 tx=0x7f63f400bdf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.896+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f63e0064730 2021-03-09T08:55:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.897+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.897+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.897+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:00.913 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:00.913 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:00.913 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:00.913 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:00.914 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:00.914 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:00.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4048410 con 0x7f63e0064730 2021-03-09T08:55:00.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.898+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:00.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.898+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:55:00.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:00.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402a1f0 con 0x7f63e0064730 2021-03-09T08:55:01.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:01.388+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:01.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:01.388+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:02.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:02.388+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:02.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:02.388+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:02.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:02.388+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:03.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:03.388+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:03.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:03.388+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:04.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:04.388+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:04.388+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:04.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:04.415+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:05.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.388+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:05.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.388+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:05.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:05.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:05.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ffc0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f63f4005f10 tx=0x7f63f400bdf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:05.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ffc0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f63f4005f10 tx=0x7f63f400bdf0).stop 2021-03-09T08:55:05.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:05.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:05.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001ffc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:05.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ffc0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:05.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:55:05.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:05.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:05.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:05.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:05.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:05.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:05.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:05.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:55:05.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:05.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f6400145bd0 2021-03-09T08:55:05.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:05.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d1e0 con 0x7f6400146050 2021-03-09T08:55:05.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:05.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:05.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:05.902 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2640 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:05.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:05.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.895+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:05.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.896+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:05.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.896+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:55:05.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.896+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:05.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.896+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:05.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:05.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:05.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:05.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:05.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:05.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:05.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:05.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:05.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:55:05.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:55:05.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:05.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:05.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015bc0 con 0x7f6400146050 2021-03-09T08:55:05.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:05.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:05.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029250 con 0x7f6400146050 2021-03-09T08:55:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:55:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.897+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc076210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.899+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:55:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.899+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:05.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T08:55:05.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.900+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:05.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.900+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:05.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.900+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:05.910 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:05.911 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:05.911 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:05.911 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:05.911 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:05.911 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:05.912 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:05.912 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:05.912 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:05.912 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:05.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T08:55:05.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.901+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:05.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.901+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:55:05.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:05.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc05b170 con 0x7f6400146050 2021-03-09T08:55:06.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:06.388+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:06.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:06.388+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:07.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:07.389+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:07.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:07.389+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:08.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:08.389+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:08.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:08.389+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:09.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:09.389+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:09.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:09.389+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:09.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:09.416+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:10.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.389+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:10.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.389+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:10.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.895+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.895+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.895+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc076210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:10.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc076210).stop 2021-03-09T08:55:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:10.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:55:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:10.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:10.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0021d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001aa80 con 0x7f63e00070a0 2021-03-09T08:55:10.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f6400145bd0 2021-03-09T08:55:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:10.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63e0064730 2021-03-09T08:55:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:10.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:10.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:10.905 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2687 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:10.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.896+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0021d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:55:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:10.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:55:10.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:55:10.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:10.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:10.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e0a0 con 0x7f63e0064730 2021-03-09T08:55:10.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:10.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:10.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T08:55:10.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:55:10.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.897+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:10.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.898+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0021d50 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7f63f003eb70 tx=0x7f63f003ecc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:10.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.898+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:55:10.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.898+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:10.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0053170 con 0x7f63e0064730 2021-03-09T08:55:10.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.899+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:10.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.899+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:10.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.899+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:10.917 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:10.917 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:10.917 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:10.918 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:10.918 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:10.918 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:10.918 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:10.918 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:10.919 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:10.919 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:10.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002d4a0 con 0x7f63e0064730 2021-03-09T08:55:10.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.899+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:10.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.899+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:55:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:10.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:55:11.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:11.389+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:11.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:11.389+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:12.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:12.389+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:12.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:12.389+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:12.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:12.389+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:13.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:13.389+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:13.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:13.389+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:14.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:14.389+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:14.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:14.389+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:14.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:14.417+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:15.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:15.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:15.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0021d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:15.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0021d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:15.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0021d50 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7f63f003eb70 tx=0x7f63f003ecc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:15.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0021d50 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7f63f003eb70 tx=0x7f63f003ecc0).stop 2021-03-09T08:55:15.899 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:15.897+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2719 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:15.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0021d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:15.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0021d50 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:15.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:55:15.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:15.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:15.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:15.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018340 con 0x7f63e00070a0 2021-03-09T08:55:15.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:15.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:55:15.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:15.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025cb0 con 0x7f6400146050 2021-03-09T08:55:15.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.896+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:15.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.897+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:15.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.897+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:15.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.897+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:15.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.897+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:15.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.897+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:15.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.897+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:55:15.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.897+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:15.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.897+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:15.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.898+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:15.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.898+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:15.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.898+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:15.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:15.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:15.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:15.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:15.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:15.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:55:15.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:55:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T08:55:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:15.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015bc0 con 0x7f6400146050 2021-03-09T08:55:15.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:55:15.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:15.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc01b1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:15.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:55:15.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.899+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:15.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc006480 con 0x7f6400146050 2021-03-09T08:55:15.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.901+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:15.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.901+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:15.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.901+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:15.917 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:15.917 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:15.918 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:15.918 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:15.918 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:15.918 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:15.920 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:15.920 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:15.920 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:15.920 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc006480 con 0x7f6400146050 2021-03-09T08:55:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.904+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.905+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:55:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:15.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc05b170 con 0x7f6400146050 2021-03-09T08:55:16.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:16.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:16.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:16.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:17.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:17.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:17.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:17.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:18.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:18.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:19.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:19.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:19.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:19.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:19.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:19.418+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:20.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:20.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc01b1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:20.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc01b1d0).stop 2021-03-09T08:55:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:20.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:20.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:55:20.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:20.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:20.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:20.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:20.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:20.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:20.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:20.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63e00070a0 2021-03-09T08:55:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017070 con 0x7f6400145bd0 2021-03-09T08:55:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:55:20.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.897+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:20.904 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2759 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:20.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:20.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:20.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:20.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:20.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:20.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:55:20.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:20.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:20.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:20.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:20.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.898+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:20.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:20.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:20.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:20.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:20.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:20.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:55:20.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:55:20.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:20.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:20.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b980 con 0x7f63e0064730 2021-03-09T08:55:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e0a0 con 0x7f63e0064730 2021-03-09T08:55:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:55:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.899+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.900+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:55:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.900+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008650 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f63f00252d0 tx=0x7f63f0025110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.900+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003eeb0 con 0x7f63e0064730 2021-03-09T08:55:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.903+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:20.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.903+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:20.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.903+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:20.914 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:20.914 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:20.915 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:20.915 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:20.915 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:20.915 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:20.915 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:20.916 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:20.916 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:20.916 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:20.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f63e0064730 2021-03-09T08:55:20.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.903+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:20.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a030 con 0x7f63e0064730 2021-03-09T08:55:20.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:20.903+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:55:21.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:21.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:21.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:21.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:22.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:22.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:22.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:22.391+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:23.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:23.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:23.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:23.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:24.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:24.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:24.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:24.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:24.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:24.418+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:25.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:25.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:25.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008650 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f63f00252d0 tx=0x7f63f0025110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008650 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 rx=0x7f63f00252d0 tx=0x7f63f0025110).stop 2021-03-09T08:55:25.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0008650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008650 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:25.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:55:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:25.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001aa80 con 0x7f63e00070a0 2021-03-09T08:55:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:25.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f6400145bd0 2021-03-09T08:55:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f950 con 0x7f6400146050 2021-03-09T08:55:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:25.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:25.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:25.905 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:25.897+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2794 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.898+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.898+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.898+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:55:25.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.898+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:25.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.898+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:25.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:25.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:25.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:25.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:25.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:25.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:25.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:25.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:25.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:55:25.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:55:25.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:25.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:25.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029250 con 0x7f6400146050 2021-03-09T08:55:25.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:25.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:25.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T08:55:25.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:55:25.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.899+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:25.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.900+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f63fc0761e0 tx=0x7f63fc06c200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.900+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:55:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.900+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T08:55:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.902+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.902+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.902+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:25.917 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:25.917 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:25.917 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:25.917 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:25.917 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:25.919 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T08:55:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.902+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.902+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:55:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:25.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f6400146050 2021-03-09T08:55:26.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:26.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:26.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:26.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:27.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:27.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:27.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:27.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:28.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:28.390+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:28.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:28.390+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:29.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:29.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:29.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:29.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:29.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:29.419+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:30.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:30.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:30.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:30.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:30.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f63fc0761e0 tx=0x7f63fc06c200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:30.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7f63fc0761e0 tx=0x7f63fc06c200).stop 2021-03-09T08:55:30.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:30.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:30.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:30.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:30.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:55:30.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:30.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:30.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0009010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:30.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:30.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:30.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:30.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:30.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00024f0 con 0x7f63e00070a0 2021-03-09T08:55:30.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:30.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002dec0 con 0x7f6400145bd0 2021-03-09T08:55:30.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:30.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010f60 con 0x7f63e0064730 2021-03-09T08:55:30.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:30.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:30.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:30.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:30.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:30.911 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:30.898+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2834 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0009010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.899+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:55:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.899+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.899+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:55:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:55:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:30.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014400 con 0x7f63e0064730 2021-03-09T08:55:30.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.900+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:30.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.901+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:30.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.901+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029250 con 0x7f63e0064730 2021-03-09T08:55:30.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.901+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:55:30.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.901+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:30.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.901+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:55:30.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.901+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:30.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.901+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0009010 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc019950).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T08:55:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.902+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.902+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.902+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:30.922 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T08:55:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.902+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.902+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:55:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:30.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049910 con 0x7f63e0064730 2021-03-09T08:55:31.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:31.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:31.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:31.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:32.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:32.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:32.391+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:33.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:33.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:33.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:33.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:34.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:34.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:34.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:34.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:34.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:34.420+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:35.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:35.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:35.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0009010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0009010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0009010 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc019950).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0009010 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc019950).stop 2021-03-09T08:55:35.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0009010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0009010 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:35.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:55:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:35.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:35.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:35.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:35.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018340 con 0x7f63e00070a0 2021-03-09T08:55:35.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:35.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:55:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cc10 con 0x7f6400146050 2021-03-09T08:55:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:35.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:35.907 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2865 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:35.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:35.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:55:35.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:35.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:35.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:35.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.899+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:35.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:35.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:35.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:35.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:35.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:35.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:35.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:35.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:35.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:55:35.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:55:35.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:35.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:35.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028720 con 0x7f6400146050 2021-03-09T08:55:35.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:35.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:35.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f3e0 con 0x7f6400146050 2021-03-09T08:55:35.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:55:35.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.900+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:35.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.901+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f63f4063000 tx=0x7f63f4047b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:35.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.901+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:55:35.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.901+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:35.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f6400146050 2021-03-09T08:55:35.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.902+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:35.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.902+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.902+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:35.918 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:35.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f6400146050 2021-03-09T08:55:35.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.902+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:35.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.902+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:55:35.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:35.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:55:36.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:36.391+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:36.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:36.391+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:37.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:37.392+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:37.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:37.392+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:38.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:38.392+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:38.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:38.392+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:39.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:39.392+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:39.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:39.392+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:39.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:39.421+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:40.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.392+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:40.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.392+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:40.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f63f4063000 tx=0x7f63f4047b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f63f4063000 tx=0x7f63f4047b20).stop 2021-03-09T08:55:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:40.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:40.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:55:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00235b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:40.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:40.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:40.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:40.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63e00070a0 2021-03-09T08:55:40.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d3e0 con 0x7f6400145bd0 2021-03-09T08:55:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002dec0 con 0x7f63e0064730 2021-03-09T08:55:40.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:40.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:40.908 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2918 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:40.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:40.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:40.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00235b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:40.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:55:40.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:40.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.900+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:40.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.901+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:40.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.901+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:40.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.901+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:40.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:40.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:40.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:40.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:40.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:40.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:55:40.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:55:40.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:40.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:40.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T08:55:40.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:40.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:40.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014400 con 0x7f63e0064730 2021-03-09T08:55:40.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:55:40.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:40.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:55:40.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00235b0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc047d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:40.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.902+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:40.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02a120 con 0x7f63e0064730 2021-03-09T08:55:40.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.903+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:40.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.903+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:40.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.903+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:40.918 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:40.918 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:40.920 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:40.920 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:40.921 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:40.921 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:40.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc074e90 con 0x7f63e0064730 2021-03-09T08:55:40.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.903+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:40.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.903+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:55:40.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:40.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc019c40 con 0x7f63e0064730 2021-03-09T08:55:41.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:41.392+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:41.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:41.392+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:42.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:42.392+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:42.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:42.392+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:42.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:42.392+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:43.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:43.392+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:43.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:43.392+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:44.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:44.393+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:44.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:44.393+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:44.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:44.422+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:45.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.393+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:45.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.393+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:45.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.900+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00235b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.900+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00235b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.900+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00235b0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc047d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:45.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.900+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00235b0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc047d30).stop 2021-03-09T08:55:45.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.900+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:45.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.900+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:45.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00235b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:45.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.900+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00235b0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:45.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:55:45.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:45.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:45.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:45.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:45.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:45.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:45.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:45.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a290 con 0x7f63e00070a0 2021-03-09T08:55:45.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:45.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:55:45.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:45.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cc10 con 0x7f6400146050 2021-03-09T08:55:45.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:45.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:45.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:45.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:45.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:45.916 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:45.901+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2946 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:55:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:45.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:45.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:45.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:45.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:45.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:55:45.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:55:45.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:45.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:45.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014270 con 0x7f6400146050 2021-03-09T08:55:45.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:45.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:45.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028720 con 0x7f6400146050 2021-03-09T08:55:45.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:55:45.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.902+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:45.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.903+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f63f40687e0 tx=0x7f63f40681e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:45.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.903+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:55:45.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.903+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:45.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a040 con 0x7f6400146050 2021-03-09T08:55:45.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.904+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:45.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.904+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:45.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.904+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:45.925 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:45.925 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:45.926 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:45.926 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:45.926 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:45.927 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f6400146050 2021-03-09T08:55:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.906+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.906+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:55:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:45.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:55:46.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:46.393+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:46.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:46.393+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:47.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:47.393+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:47.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:47.393+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:48.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:48.393+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:48.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:48.393+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:49.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:49.393+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:49.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:49.393+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:49.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:49.423+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:50.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.393+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:50.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.393+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:55:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f63f40687e0 tx=0x7f63f40681e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7f63f40687e0 tx=0x7f63f40681e0).stop 2021-03-09T08:55:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c4a0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:55:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00292a0 con 0x7f63e00070a0 2021-03-09T08:55:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021600 con 0x7f6400145bd0 2021-03-09T08:55:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002dec0 con 0x7f63e0064730 2021-03-09T08:55:50.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:50.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:50.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:50.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:50.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:50.911 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:50.902+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 2994 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.903+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.903+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:55:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.903+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:50.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.903+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:50.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.903+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:50.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.903+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.903+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:50.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:50.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:50.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:55:50.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:55:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a820 con 0x7f63e0064730 2021-03-09T08:55:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:50.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:50.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b980 con 0x7f63e0064730 2021-03-09T08:55:50.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:55:50.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.904+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:50.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.905+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f63e006b990 tx=0x7f63f0006910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.905+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:55:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.905+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f63e0064730 2021-03-09T08:55:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.906+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:50.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.906+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:50.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.906+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:50.920 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:50.921 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:50.921 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:50.921 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:50.921 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:50.922 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:50.922 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:50.922 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:50.922 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:50.922 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:50.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0064730 2021-03-09T08:55:50.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.909+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:50.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.909+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:55:50.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:50.911+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:55:51.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:51.393+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:51.393+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:52.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:52.393+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:52.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:52.393+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:52.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:52.393+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:53.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:53.394+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:53.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:53.394+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:54.394+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:54.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:54.394+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:54.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:54.424+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:55:55.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.394+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:55.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.394+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:55.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:55:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:55:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f63e006b990 tx=0x7f63f0006910).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:55:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f63e006b990 tx=0x7f63f0006910).stop 2021-03-09T08:55:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:55.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:55:55.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:55:55.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:55:55.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:55:55.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:55:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:55:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:55:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:55:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d3e0 con 0x7f63e00070a0 2021-03-09T08:55:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018340 con 0x7f6400145bd0 2021-03-09T08:55:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:55:55.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:55:55.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:55:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:55:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:55:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:55:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:55:55.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.904+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:55:55.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.904+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:55:55.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.904+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:55:55.913 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:55:55.903+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3018 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:55:55.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.904+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:55:55.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.904+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:55:55.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.904+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:55:55.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:55:55.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:55.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:55.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:55:55.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:55:55.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:55:55.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:55:55.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:55:55.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:55.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T08:55:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:55:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:55:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T08:55:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:55:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:55:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:55:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.905+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c3c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7f63fc0517d0 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:55:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0513e0 con 0x7f6400146050 2021-03-09T08:55:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.906+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:55:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.906+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:55:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.906+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:55:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:55:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:55:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:55:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:55:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:55:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:55:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:55:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:55:55.922 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:55:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f6400146050 2021-03-09T08:55:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.907+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:55:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.907+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:55:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:55.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T08:55:56.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:56.394+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:56.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:56.394+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:57.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:57.394+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:57.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:57.394+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:58.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:58.394+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:58.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:58.394+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:59.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:59.395+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:55:59.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:59.395+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:55:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:55:59.425+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.395+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:00.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.395+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:00.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c3c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7f63fc0517d0 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c3c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 rx=0x7f63fc0517d0 tx=0x7f63fc003a50).stop 2021-03-09T08:56:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:00.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:00.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:00.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c3c0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:00.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:56:00.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:00.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:00.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:00.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:00.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:00.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cb70 con 0x7f63e00070a0 2021-03-09T08:56:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021600 con 0x7f6400145bd0 2021-03-09T08:56:00.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008a30 con 0x7f63e0064730 2021-03-09T08:56:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:00.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:56:00.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:00.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:00.913 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:00.904+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3062 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:00.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.905+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:00.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.905+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:00.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.905+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:00.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:00.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:00.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:00.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:00.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:00.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:56:00.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:56:00.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:00.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:00.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014400 con 0x7f63e0064730 2021-03-09T08:56:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a820 con 0x7f63e0064730 2021-03-09T08:56:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:56:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f0021db0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:56:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.906+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0054420 con 0x7f63e0064730 2021-03-09T08:56:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.907+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.907+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.907+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:00.922 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0064730 2021-03-09T08:56:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.907+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.907+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:56:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:00.908+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:56:01.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:01.395+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:01.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:01.395+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:02.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:02.395+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:02.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:02.395+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:02.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:02.395+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:03.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:03.395+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:03.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:03.395+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:04.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:04.395+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:04.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:04.395+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:04.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:04.426+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:05.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.395+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:05.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.395+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:05.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:56:05.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:05.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f0021db0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:05.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f63f000a0a0 tx=0x7f63f0021db0).stop 2021-03-09T08:56:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e5a0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:05.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:56:05.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:05.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:05.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:05.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:05.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:05.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:05.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:05.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011bd0 con 0x7f63e00070a0 2021-03-09T08:56:05.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:05.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002dec0 con 0x7f6400145bd0 2021-03-09T08:56:05.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:05.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b650 con 0x7f6400146050 2021-03-09T08:56:05.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:05.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:05.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:05.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:05.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:05.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:05.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:56:05.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:05.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:05.917 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:05.905+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3096 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:05.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:05.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:05.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:05.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:05.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:05.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:05.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:05.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:05.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:56:05.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:56:05.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:05.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:05.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T08:56:05.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:05.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:05.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014270 con 0x7f6400146050 2021-03-09T08:56:05.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.907+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:56:05.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.907+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:05.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.907+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:56:05.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.907+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132d0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f63f4067330 tx=0x7f63f40674f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:05.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.907+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:05.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f6400146050 2021-03-09T08:56:05.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.910+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:05.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.910+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:05.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.910+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:05.925 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:05.926 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:05.926 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:05.926 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:05.927 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:05.929 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40477a0 con 0x7f6400146050 2021-03-09T08:56:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.910+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.910+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:56:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:05.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:56:06.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:06.396+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:06.396+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:07.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:07.396+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:07.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:07.396+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:08.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:08.396+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:08.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:08.396+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:09.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:09.396+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:09.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:09.396+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:09.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:09.427+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:10.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.396+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:10.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.396+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:10.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00132d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:56:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00132d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132d0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f63f4067330 tx=0x7f63f40674f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132d0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f63f4067330 tx=0x7f63f40674f0).stop 2021-03-09T08:56:10.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00132d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132d0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:10.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:56:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:10.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:10.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:10.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:10.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024fe0 con 0x7f63e00070a0 2021-03-09T08:56:10.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:10.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018340 con 0x7f6400145bd0 2021-03-09T08:56:10.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:10.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b710 con 0x7f63e0064730 2021-03-09T08:56:10.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:10.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:10.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:10.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:10.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:10.915 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:10.906+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3135 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:10.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.907+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:10.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.907+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:56:10.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.907+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:10.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.907+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:10.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.907+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:10.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.907+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:10.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.907+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:10.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:10.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:10.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:56:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:56:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:10.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:10.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019a40 con 0x7f63e0064730 2021-03-09T08:56:10.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:10.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T08:56:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:56:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.908+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f690 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f63fc05cde0 tx=0x7f63fc02c300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.909+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:56:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.909+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e0064730 2021-03-09T08:56:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.910+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.910+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.910+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:10.926 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:10.926 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:10.926 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:10.926 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:10.926 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:10.927 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:10.927 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:10.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc006480 con 0x7f63e0064730 2021-03-09T08:56:10.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.910+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:10.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.910+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:56:10.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:10.911+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e0064730 2021-03-09T08:56:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:11.396+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:11.396+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:12.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:12.396+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:12.396+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:12.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:12.396+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:13.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:13.396+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:13.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:13.396+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:14.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:14.397+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:14.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:14.397+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:14.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:14.428+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:15.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.397+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:15.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.397+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:15.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:15.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:15.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f690 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f63fc05cde0 tx=0x7f63fc02c300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f690 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7f63fc05cde0 tx=0x7f63fc02c300).stop 2021-03-09T08:56:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:15.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000f690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:15.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f690 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:15.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:56:15.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:15.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:15.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:15.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:15.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:15.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:15.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:15.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b650 con 0x7f63e00070a0 2021-03-09T08:56:15.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:15.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00184a0 con 0x7f6400145bd0 2021-03-09T08:56:15.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:15.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078730 con 0x7f6400146050 2021-03-09T08:56:15.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:15.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:15.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:15.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:15.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:15.917 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3166 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:15.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:15.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:56:15.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:15.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.907+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:15.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:15.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:15.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:15.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:15.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:56:15.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:56:15.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:15.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:15.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014270 con 0x7f6400146050 2021-03-09T08:56:15.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:15.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:15.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T08:56:15.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:56:15.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.908+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:15.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.909+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:56:15.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.909+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e9a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40216b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:15.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.909+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:15.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a040 con 0x7f6400146050 2021-03-09T08:56:15.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.910+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.910+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.910+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:15.926 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:15.926 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:15.926 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:15.926 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:15.926 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:15.927 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:15.927 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:15.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T08:56:15.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.910+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:15.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.910+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:56:15.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:15.911+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:56:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:16.397+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:16.397+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:17.396+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:17.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:17.396+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:18.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:18.397+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:18.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:18.397+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:19.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:19.397+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:19.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:19.397+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:19.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:19.429+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:20.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.397+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:20.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.397+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:20.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:20.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:20.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e9a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40216b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:20.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e9a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40216b0).stop 2021-03-09T08:56:20.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:20.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e9a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e9a0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:56:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:20.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:20.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:20.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e170 con 0x7f63e00070a0 2021-03-09T08:56:20.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:20.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00294a0 con 0x7f6400145bd0 2021-03-09T08:56:20.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:20.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:56:20.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:20.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:20.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:20.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:20.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:20.917 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:20.908+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3206 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:20.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.909+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:20.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.909+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:56:20.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.909+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:20.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.909+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:20.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.909+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:20.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.909+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:20.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.909+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:20.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:20.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:20.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:20.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:20.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:20.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:56:20.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:56:20.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:20.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T08:56:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014270 con 0x7f63e0064730 2021-03-09T08:56:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:56:20.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:20.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:56:20.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:20.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028160 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f63f40672d0 tx=0x7f63f4047ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f63e0064730 2021-03-09T08:56:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.911+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.911+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.911+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:20.927 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:20.927 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:20.927 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:20.927 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:20.928 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:20.928 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:20.928 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:20.928 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:20.928 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:20.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.911+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4062ec0 con 0x7f63e0064730 2021-03-09T08:56:20.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.911+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:20.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.911+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:56:20.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:20.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f63e0064730 2021-03-09T08:56:21.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:21.397+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:21.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:21.397+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:22.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:22.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:22.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:22.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:22.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:22.398+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:23.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:23.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:23.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:23.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:24.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:24.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:24.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:24.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:24.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:24.430+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:25.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:25.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:25.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0028160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:25.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0028160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:25.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028160 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f63f40672d0 tx=0x7f63f4047ab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:25.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028160 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7f63f40672d0 tx=0x7f63f4047ab0).stop 2021-03-09T08:56:25.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:25.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:25.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0028160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:25.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0028160 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:25.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:56:25.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:25.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:25.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:25.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b710 con 0x7f63e00070a0 2021-03-09T08:56:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f020 con 0x7f6400145bd0 2021-03-09T08:56:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:25.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f6400146050 2021-03-09T08:56:25.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:25.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:25.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:56:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:25.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:25.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:25.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:25.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:25.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:25.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:25.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:25.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:56:25.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:56:25.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:25.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:25.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00180d0 con 0x7f6400146050 2021-03-09T08:56:25.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:25.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:25.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019a40 con 0x7f6400146050 2021-03-09T08:56:25.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:56:25.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:25.924 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:25.909+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3242 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:25.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:56:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0242b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.910+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0641a0 con 0x7f6400146050 2021-03-09T08:56:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.912+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:25.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.912+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:25.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.912+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:25.926 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:25.928 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:25.928 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:25.928 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:25.929 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:25.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06ce70 con 0x7f6400146050 2021-03-09T08:56:25.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.912+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:25.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.912+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:56:25.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:25.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f6400146050 2021-03-09T08:56:26.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:26.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:26.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:26.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:27.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:27.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:27.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:27.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:28.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:28.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:28.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:28.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:29.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:29.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:29.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:29.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:29.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:29.431+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:30.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:30.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:30.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0016af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:30.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0016af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:30.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0242b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:30.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0242b0).stop 2021-03-09T08:56:30.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0016af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:56:30.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:30.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a870 con 0x7f63e00070a0 2021-03-09T08:56:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:30.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b7b0 con 0x7f6400145bd0 2021-03-09T08:56:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:56:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.909+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:30.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.910+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:30.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.910+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:30.918 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:30.910+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3281 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.910+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:30.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.911+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.911+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:56:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.911+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.911+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:30.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:30.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:56:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:56:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:30.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:30.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d3e0 con 0x7f63e0064730 2021-03-09T08:56:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00180d0 con 0x7f63e0064730 2021-03-09T08:56:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:56:30.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.912+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:30.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.913+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7f63fc064770 tx=0x7f63fc023ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:30.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.913+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:56:30.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.913+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:30.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0647e0 con 0x7f63e0064730 2021-03-09T08:56:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.914+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.914+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.914+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:30.929 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:30.929 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:30.929 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:30.929 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:30.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06ce70 con 0x7f63e0064730 2021-03-09T08:56:30.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.914+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:30.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.914+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:56:30.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:30.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049910 con 0x7f63e0064730 2021-03-09T08:56:31.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:31.399+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:31.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:31.399+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:32.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:32.399+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:32.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:32.399+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:32.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:32.399+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:33.398+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:33.398+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:34.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:34.399+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:34.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:34.399+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:34.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:34.432+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:35.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.399+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:35.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.399+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:35.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:35.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:35.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7f63fc064770 tx=0x7f63fc023ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:35.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 rx=0x7f63fc064770 tx=0x7f63fc023ec0).stop 2021-03-09T08:56:35.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:35.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:35.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007d220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:35.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:35.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:56:35.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:35.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.910+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:35.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078730 con 0x7f63e00070a0 2021-03-09T08:56:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b650 con 0x7f6400145bd0 2021-03-09T08:56:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:35.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:56:35.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:35.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:35.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:35.919 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3322 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:35.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:35.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:35.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:35.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:56:35.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:35.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.911+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:35.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.912+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:35.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.912+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:35.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.912+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:35.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:35.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:35.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:35.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:35.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:35.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:56:35.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:56:35.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:35.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:35.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014270 con 0x7f6400146050 2021-03-09T08:56:35.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:35.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:35.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d3e0 con 0x7f6400146050 2021-03-09T08:56:35.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:56:35.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.913+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:35.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.914+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:56:35.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.914+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:35.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc051200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:35.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc05b380 con 0x7f6400146050 2021-03-09T08:56:35.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.916+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:35.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.916+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:35.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.916+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:35.931 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:35.932 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:35.932 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:35.932 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:35.932 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:35.933 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:35.933 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:35.933 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:35.933 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:35.934 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:35.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0647e0 con 0x7f6400146050 2021-03-09T08:56:35.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.917+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:35.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049910 con 0x7f6400146050 2021-03-09T08:56:35.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:35.918+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:56:36.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:36.399+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:36.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:36.399+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:37.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:37.399+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:37.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:37.399+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:38.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:38.400+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:38.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:38.400+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:39.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:39.400+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:39.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:39.400+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:39.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:39.433+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:40.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.400+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:40.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.400+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:40.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0016af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:40.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0016af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:40.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc051200).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:40.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc051200).stop 2021-03-09T08:56:40.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:40.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:40.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0016af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:40.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:40.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:56:40.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:40.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:40.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:40.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012c40 con 0x7f63e00070a0 2021-03-09T08:56:40.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:40.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a870 con 0x7f6400145bd0 2021-03-09T08:56:40.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:40.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010540 con 0x7f63e0064730 2021-03-09T08:56:40.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:40.921 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3363 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:40.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:40.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:40.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:56:40.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:40.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:40.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:40.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:40.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:40.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:40.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:40.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.912+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:40.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:40.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:40.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:40.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:56:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:56:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:40.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011110 con 0x7f63e0064730 2021-03-09T08:56:40.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:40.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:40.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014400 con 0x7f63e0064730 2021-03-09T08:56:40.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:56:40.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:40.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f63f00265a0 tx=0x7f63f0026290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:40.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:56:40.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.913+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:40.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f63e0064730 2021-03-09T08:56:40.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.914+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:40.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.914+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:40.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.914+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:40.931 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:40.931 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:40.931 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:40.931 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:40.932 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:40.932 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:40.932 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:40.932 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:40.933 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:40.933 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:40.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f63e0064730 2021-03-09T08:56:40.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.914+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:40.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.914+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:56:40.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:40.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a030 con 0x7f63e0064730 2021-03-09T08:56:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:41.400+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:41.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:41.400+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:42.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:42.400+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:42.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:42.400+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:42.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:42.400+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:43.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:43.400+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:43.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:43.400+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:44.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:44.401+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:44.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:44.401+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:44.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:44.434+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:45.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.400+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:45.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.400+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:45.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:45.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:45.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f63f00265a0 tx=0x7f63f0026290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:45.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f63f00265a0 tx=0x7f63f0026290).stop 2021-03-09T08:56:45.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:45.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007d220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d220 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:56:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:45.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:45.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078730 con 0x7f63e00070a0 2021-03-09T08:56:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:45.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012c40 con 0x7f6400145bd0 2021-03-09T08:56:45.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:45.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f6400146050 2021-03-09T08:56:45.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:45.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:45.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:45.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:45.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:45.922 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3398 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:45.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:45.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:56:45.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:45.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:45.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:45.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:45.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.913+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:45.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:45.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:45.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:45.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:45.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:56:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:56:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f900 con 0x7f6400146050 2021-03-09T08:56:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T08:56:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:56:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.914+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:45.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.915+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:56:45.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f63f40525c0 tx=0x7f63f4003ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:45.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.915+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:45.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f6400146050 2021-03-09T08:56:45.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.917+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:45.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.917+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:45.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.917+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:45.930 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:45.931 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:45.931 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:45.931 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:45.931 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:45.932 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:45.932 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:45.932 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:45.932 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:45.932 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:45.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f6400146050 2021-03-09T08:56:45.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.919+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:45.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.919+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:56:45.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:45.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40213a0 con 0x7f6400146050 2021-03-09T08:56:46.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:46.401+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:46.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:46.401+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:47.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:47.401+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:47.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:47.401+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:48.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:48.401+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:48.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:48.401+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:49.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:49.401+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:49.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:49.401+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:49.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:49.435+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:50.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.401+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:50.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.401+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:50.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0016af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:50.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0016af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f63f40525c0 tx=0x7f63f4003ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f63f40525c0 tx=0x7f63f4003ad0).stop 2021-03-09T08:56:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:50.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0016af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:50.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0016af0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:50.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:56:50.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:50.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:50.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:50.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e00070a0 2021-03-09T08:56:50.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:50.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001dc00 con 0x7f6400145bd0 2021-03-09T08:56:50.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:50.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021800 con 0x7f63e0064730 2021-03-09T08:56:50.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:50.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:50.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:50.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:50.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:50.922 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3442 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:50.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:50.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:56:50.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:50.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:50.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:50.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:50.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.914+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:50.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:50.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:50.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:50.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:50.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:50.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:56:50.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:56:50.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:50.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:50.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014400 con 0x7f63e0064730 2021-03-09T08:56:50.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:50.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:50.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f900 con 0x7f63e0064730 2021-03-09T08:56:50.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:56:50.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.915+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:50.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.916+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:56:50.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.916+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cc60 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f400bd20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:50.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.916+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:50.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405a070 con 0x7f63e0064730 2021-03-09T08:56:50.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.917+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:50.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.917+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:50.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.917+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:50.929 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:50.930 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:50.930 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:50.930 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:50.930 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:50.931 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:50.931 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:50.931 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:50.931 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:50.931 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40476d0 con 0x7f63e0064730 2021-03-09T08:56:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.917+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.917+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:56:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:50.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T08:56:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:51.401+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:51.401+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:52.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:52.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:52.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:52.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:52.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:52.402+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:53.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:53.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:53.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:53.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:54.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:54.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:54.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:54.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:54.436+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:56:55.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.401+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:55.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.401+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:55.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:56:55.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:56:55.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cc60 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f400bd20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:56:55.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cc60 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f400bd20).stop 2021-03-09T08:56:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:56:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001cc60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:56:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001cc60 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:56:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:56:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:56:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:56:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:56:55.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:56:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001fb50 con 0x7f63e00070a0 2021-03-09T08:56:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:55.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:56:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:56:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078730 con 0x7f6400146050 2021-03-09T08:56:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:56:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:56:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:56:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:56:55.924 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3478 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:56:55.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:56:55.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:56:55.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:56:55.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:56:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:56:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:56:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:56:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:56:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:56:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:56:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:56:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:56:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:56:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T08:56:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:56:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:56:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014400 con 0x7f6400146050 2021-03-09T08:56:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:56:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.915+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.917+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015a30 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7f63f40616b0 tx=0x7f63f4029180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:56:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.917+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:56:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.917+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:56:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f6400146050 2021-03-09T08:56:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.917+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:56:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.917+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:56:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.917+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:56:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:56:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:56:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:56:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:56:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:56:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:56:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:56:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:56:55.934 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:56:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40295e0 con 0x7f6400146050 2021-03-09T08:56:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.917+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:56:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.918+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:56:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:55.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:56:56.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:56.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:56.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:56.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:57.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:57.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:57.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:57.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:58.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:58.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:58.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:58.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:59.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:59.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:56:59.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:59.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:56:59.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:56:59.437+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:00.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:00.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:57:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015a30 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7f63f40616b0 tx=0x7f63f4029180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015a30 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7f63f40616b0 tx=0x7f63f4029180).stop 2021-03-09T08:57:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015a30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015a30 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:57:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cc10 con 0x7f63e00070a0 2021-03-09T08:57:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f020 con 0x7f6400145bd0 2021-03-09T08:57:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027000 con 0x7f63e0064730 2021-03-09T08:57:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:00.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:00.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:00.924 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3505 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:00.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:00.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:57:00.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:00.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:00.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:00.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:00.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:00.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:00.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:00.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:57:00.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:57:00.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:00.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:00.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014400 con 0x7f63e0064730 2021-03-09T08:57:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014270 con 0x7f63e0064730 2021-03-09T08:57:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:57:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:57:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017470 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:00.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.917+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:00.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e0064730 2021-03-09T08:57:00.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.918+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.918+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.918+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:00.939 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:00.939 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:00.942 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0327d0 con 0x7f63e0064730 2021-03-09T08:57:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.918+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.918+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:57:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:00.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02b6e0 con 0x7f63e0064730 2021-03-09T08:57:01.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:01.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:01.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:01.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:02.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:02.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:02.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:02.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:02.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:02.403+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:03.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:03.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:03.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:03.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:04.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:04.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:04.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:04.438+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:05.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:05.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:05.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0017470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:57:05.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0017470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:05.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017470 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:05.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017470 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009990).stop 2021-03-09T08:57:05.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:05.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:05.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0017470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:05.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017470 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:05.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:57:05.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00163d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:05.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:05.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:05.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:05.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:05.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:05.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:05.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017270 con 0x7f63e00070a0 2021-03-09T08:57:05.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:05.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d980 con 0x7f6400145bd0 2021-03-09T08:57:05.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:05.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:57:05.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:05.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:05.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.916+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:05.924 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3545 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:05.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:05.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:05.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:05.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:57:05.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:05.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.917+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:57:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:57:05.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:05.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:05.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d7b0 con 0x7f6400146050 2021-03-09T08:57:05.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:05.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T08:57:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:57:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.918+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.919+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:57:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.919+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.919+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00163d0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7f63f4003ca0 tx=0x7f63f4005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c030 con 0x7f6400146050 2021-03-09T08:57:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.921+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.921+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.921+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:05.935 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4064120 con 0x7f6400146050 2021-03-09T08:57:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.924+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:57:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:05.924+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:57:06.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:06.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:06.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:06.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:07.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:07.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:07.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:07.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:08.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:08.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:08.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:08.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:09.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:09.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:09.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:09.439+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:10.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:10.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00163d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00163d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:10.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00163d0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7f63f4003ca0 tx=0x7f63f4005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:10.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00163d0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7f63f4003ca0 tx=0x7f63f4005110).stop 2021-03-09T08:57:10.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:10.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:10.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00163d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:10.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00163d0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:57:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:10.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:10.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c470 con 0x7f63e00070a0 2021-03-09T08:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b5b0 con 0x7f6400145bd0 2021-03-09T08:57:10.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce10 con 0x7f63e0064730 2021-03-09T08:57:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:10.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:10.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.917+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:10.926 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:10.918+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3583 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:10.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.918+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:10.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.918+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:57:10.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.918+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:10.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.918+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:10.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.918+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:10.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.918+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:10.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.918+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:10.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:10.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:10.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:10.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:10.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:10.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:57:10.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:57:10.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:10.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016ec0 con 0x7f63e0064730 2021-03-09T08:57:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d7b0 con 0x7f63e0064730 2021-03-09T08:57:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:57:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.919+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.920+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017e90 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f63f4047b60 tx=0x7f63f4022490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.920+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:57:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.920+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f63e0064730 2021-03-09T08:57:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.921+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.921+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.921+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:10.936 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f63e0064730 2021-03-09T08:57:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.924+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:10.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.924+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:57:10.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:10.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63e0064730 2021-03-09T08:57:11.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:11.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:11.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:11.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:12.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:12.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:12.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:12.404+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:13.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:13.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:13.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:13.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:14.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:14.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:14.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:14.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:14.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:14.440+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:15.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:15.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:15.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0017e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0017e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017e90 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f63f4047b60 tx=0x7f63f4022490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:15.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017e90 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f63f4047b60 tx=0x7f63f4022490).stop 2021-03-09T08:57:15.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:15.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:15.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0017e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:15.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017e90 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:15.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:57:15.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:15.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:15.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:15.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:15.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:15.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:15.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078730 con 0x7f63e00070a0 2021-03-09T08:57:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:57:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:15.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014070 con 0x7f6400146050 2021-03-09T08:57:15.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:15.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:15.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:15.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:15.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.918+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:15.927 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:15.919+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3620 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:15.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.919+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:15.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.919+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:57:15.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.919+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:15.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.919+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:15.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.919+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:15.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.919+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:15.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.919+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:15.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:15.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:15.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:15.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:15.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:15.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:57:15.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:57:15.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:15.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:15.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025c60 con 0x7f6400146050 2021-03-09T08:57:15.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:15.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:15.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014400 con 0x7f6400146050 2021-03-09T08:57:15.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:57:15.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.920+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:15.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc075fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.921+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:57:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.921+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T08:57:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.922+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.922+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.922+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:15.937 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:15.937 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:15.937 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:15.937 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:15.937 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:15.938 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T08:57:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.924+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.924+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:57:15.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:15.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T08:57:16.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:16.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:16.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:17.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:17.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:17.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:17.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:18.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:18.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:18.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:18.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:19.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:19.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:19.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:19.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:19.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:19.441+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:20.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:20.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:20.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:20.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:20.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc075fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:20.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc075fe0).stop 2021-03-09T08:57:20.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:20.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:20.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006c5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:57:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:20.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:20.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:20.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0020a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:20.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:20.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e00070a0 2021-03-09T08:57:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019450 con 0x7f6400145bd0 2021-03-09T08:57:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:20.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:57:20.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:20.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:20.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:20.928 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3653 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:20.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:20.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:20.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.919+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0020a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:20.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.920+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:57:20.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.920+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:20.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.920+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:20.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.920+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:20.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.920+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:20.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.920+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:20.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:20.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:20.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:20.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:20.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:20.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:57:20.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:57:20.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:20.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:20.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ae10 con 0x7f63e0064730 2021-03-09T08:57:20.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:20.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:20.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011110 con 0x7f63e0064730 2021-03-09T08:57:20.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:57:20.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:20.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:57:20.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0020a10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f005c2c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:20.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.921+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:20.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00542b0 con 0x7f63e0064730 2021-03-09T08:57:20.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.922+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:20.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.922+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:20.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.922+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:20.938 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:20.938 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:20.939 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:20.939 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:20.939 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:20.939 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:20.940 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:20.940 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:20.940 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:20.940 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:20.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00542b0 con 0x7f63e0064730 2021-03-09T08:57:20.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.922+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:20.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.922+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:57:20.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:20.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:57:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:21.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:21.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:22.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:22.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:22.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:22.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:22.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:22.406+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:23.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:23.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:23.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:23.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:24.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:24.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:24.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:24.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:24.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:24.442+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:25.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:25.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:25.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0020a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:57:25.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0020a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:25.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0020a10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f005c2c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:25.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0020a10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f005c2c0).stop 2021-03-09T08:57:25.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:25.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:25.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0020a10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:25.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0020a10 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:57:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:25.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:25.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:25.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:25.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:25.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c480 con 0x7f63e00070a0 2021-03-09T08:57:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022ed0 con 0x7f6400145bd0 2021-03-09T08:57:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d6a0 con 0x7f6400146050 2021-03-09T08:57:25.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:25.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:25.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:25.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:25.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:25.929 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:25.920+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3698 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:25.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:25.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.921+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:57:25.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.921+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:25.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.921+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:25.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.921+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:25.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.921+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:25.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.921+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:25.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:25.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:25.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:25.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:25.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:25.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:57:25.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:57:25.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:25.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:25.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024e30 con 0x7f6400146050 2021-03-09T08:57:25.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:25.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:25.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ae10 con 0x7f6400146050 2021-03-09T08:57:25.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:57:25.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:25.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:57:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7f63f0023330 tx=0x7f63f00231e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.922+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.923+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0054250 con 0x7f6400146050 2021-03-09T08:57:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.923+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.923+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:25.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.923+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:25.939 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:25.939 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:25.939 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:25.940 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:25.940 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:25.940 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:25.940 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:25.940 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:25.941 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0054250 con 0x7f6400146050 2021-03-09T08:57:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.924+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.924+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:57:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:25.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f6400146050 2021-03-09T08:57:26.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:26.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:26.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:26.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:27.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:27.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:27.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:27.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:28.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:28.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:28.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:28.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:29.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:29.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:29.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:29.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:29.443+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:30.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:30.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:30.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:30.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:30.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7f63f0023330 tx=0x7f63f00231e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:30.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 rx=0x7f63f0023330 tx=0x7f63f00231e0).stop 2021-03-09T08:57:30.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:57:30.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:30.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:30.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:30.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:30.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b450 con 0x7f63e00070a0 2021-03-09T08:57:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:30.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:57:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012aa0 con 0x7f63e0064730 2021-03-09T08:57:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:30.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:30.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:30.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:30.930 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:30.921+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3731 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:30.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.922+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:30.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.922+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:57:30.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.922+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:30.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.922+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:30.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.922+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:30.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.922+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:30.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.922+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:30.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:30.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:30.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:30.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:30.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:30.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:57:30.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:57:30.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:30.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:30.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006c5b0 con 0x7f63e0064730 2021-03-09T08:57:30.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:30.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:30.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016ec0 con 0x7f63e0064730 2021-03-09T08:57:30.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:57:30.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:30.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:57:30.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018af0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f63f403bcf0 tx=0x7f63f4051180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.923+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f63e0064730 2021-03-09T08:57:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.926+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.926+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.926+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:30.938 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:30.938 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:30.938 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:30.938 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:30.938 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:30.939 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:30.939 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0064730 2021-03-09T08:57:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.926+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4021380 con 0x7f63e0064730 2021-03-09T08:57:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:30.926+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:57:31.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:31.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:31.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:31.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:32.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:32.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:32.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:32.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:32.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:32.406+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:33.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:33.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:33.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:33.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:34.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:34.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:34.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:34.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:34.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:34.444+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:35.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:35.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:35.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0018af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:57:35.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0018af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:35.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018af0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f63f403bcf0 tx=0x7f63f4051180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:35.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018af0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7f63f403bcf0 tx=0x7f63f4051180).stop 2021-03-09T08:57:35.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:35.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:35.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0018af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:35.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018af0 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:35.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:57:35.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:35.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:35.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:35.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:35.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:35.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:35.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:35.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016d80 con 0x7f63e00070a0 2021-03-09T08:57:35.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:35.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022ed0 con 0x7f6400145bd0 2021-03-09T08:57:35.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:35.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d6a0 con 0x7f6400146050 2021-03-09T08:57:35.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:35.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:35.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:35.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:35.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:35.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:35.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:57:35.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:35.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.922+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:35.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:35.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:35.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:35.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:35.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:35.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:35.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:35.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:35.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:57:35.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:57:35.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:35.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:35.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f730 con 0x7f6400146050 2021-03-09T08:57:35.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:35.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:35.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025c60 con 0x7f6400146050 2021-03-09T08:57:35.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:57:35.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:35.937 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:35.923+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3769 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:35.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.924+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:57:35.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf90 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:35.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.924+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:35.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T08:57:35.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.924+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.924+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.924+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:35.941 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:35.941 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:35.941 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:35.941 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:35.941 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:35.942 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:35.942 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:35.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0641a0 con 0x7f6400146050 2021-03-09T08:57:35.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.926+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:35.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc046600 con 0x7f6400146050 2021-03-09T08:57:35.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:35.926+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:57:36.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:36.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:36.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:36.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:37.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:37.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:37.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:37.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:38.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:38.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:38.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:38.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:39.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:39.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:39.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:39.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:39.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:39.445+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:40.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:40.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:40.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:57:40.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:40.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf90 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf90 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).stop 2021-03-09T08:57:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006cf90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:40.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf90 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:40.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:57:40.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:40.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:40.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:40.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:40.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:40.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:40.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:40.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00203a0 con 0x7f63e00070a0 2021-03-09T08:57:40.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:40.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b060 con 0x7f6400145bd0 2021-03-09T08:57:40.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:40.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:57:40.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:40.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:40.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:40.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:40.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:40.932 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3805 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:40.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:40.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:57:40.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:40.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:40.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:40.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:40.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:40.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:40.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:40.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:40.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:40.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:40.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:57:40.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:57:40.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:40.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:40.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00106f0 con 0x7f63e0064730 2021-03-09T08:57:40.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:40.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:40.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f730 con 0x7f63e0064730 2021-03-09T08:57:40.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:57:40.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.924+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:40.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.925+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:57:40.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.925+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:40.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.925+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc0471a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:40.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.927+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d3c0 con 0x7f63e0064730 2021-03-09T08:57:40.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.927+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:40.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.927+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:40.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.927+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:40.939 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:40.939 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:40.940 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:40.940 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:40.940 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:40.940 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:40.940 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:40.941 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:40.941 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:40.941 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:40.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d3c0 con 0x7f63e0064730 2021-03-09T08:57:40.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.928+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:40.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.928+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:57:40.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:40.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049910 con 0x7f63e0064730 2021-03-09T08:57:41.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:41.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:41.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:41.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:42.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:42.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:42.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:42.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:42.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:42.408+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:43.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:43.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:43.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:43.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:44.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:44.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:44.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:44.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:44.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:44.446+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:45.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:45.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:45.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:57:45.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:45.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc0471a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:45.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc0471a0).stop 2021-03-09T08:57:45.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:45.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:45.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:45.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:45.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:57:45.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:45.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:45.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0028eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:45.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:45.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:45.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:45.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:45.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f63e00070a0 2021-03-09T08:57:45.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:45.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014810 con 0x7f6400145bd0 2021-03-09T08:57:45.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:45.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014450 con 0x7f6400146050 2021-03-09T08:57:45.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:45.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:45.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:45.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:45.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:45.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0028eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:45.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:57:45.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:45.939 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3842 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:45.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:45.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:45.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:45.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:45.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:45.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:45.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:45.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:45.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:45.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:57:45.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:57:45.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:45.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:45.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001ae20 con 0x7f6400146050 2021-03-09T08:57:45.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:45.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:45.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00106f0 con 0x7f6400146050 2021-03-09T08:57:45.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:57:45.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.925+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:45.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.926+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0028eb0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc076130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:45.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.926+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:57:45.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.926+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:45.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.927+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f6400146050 2021-03-09T08:57:45.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.927+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:45.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.927+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:45.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.927+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:45.951 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:45.951 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:45.952 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:45.952 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:45.954 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:45.955 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:45.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f6400146050 2021-03-09T08:57:45.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.928+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:45.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.928+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:57:45.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:45.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049910 con 0x7f6400146050 2021-03-09T08:57:46.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:46.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:46.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:46.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:47.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:47.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:47.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:47.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:48.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:48.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:48.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:48.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:49.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:49.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:49.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:49.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:49.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:49.447+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:50.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:50.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:50.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0028eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:57:50.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0028eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:50.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0028eb0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc076130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:50.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0028eb0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc076130).stop 2021-03-09T08:57:50.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:50.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:50.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0028eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:50.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0028eb0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:50.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:57:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:50.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:50.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:50.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:50.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:50.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:50.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024790 con 0x7f63e00070a0 2021-03-09T08:57:50.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:50.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012e40 con 0x7f6400145bd0 2021-03-09T08:57:50.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:50.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021180 con 0x7f63e0064730 2021-03-09T08:57:50.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:50.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:50.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.926+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:50.938 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:50.927+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3881 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:50.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.927+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:50.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.927+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:50.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:50.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.928+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:57:50.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.928+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:50.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.928+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:50.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.928+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:50.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.928+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:50.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.928+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:50.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:50.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:50.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:50.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:57:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:57:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015370 con 0x7f63e0064730 2021-03-09T08:57:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024e30 con 0x7f63e0064730 2021-03-09T08:57:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:57:50.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.930+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:50.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006a9b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f63f0065630 tx=0x7f63f0054250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:50.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.931+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:57:50.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.931+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:50.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003f630 con 0x7f63e0064730 2021-03-09T08:57:50.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.932+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:50.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.932+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:50.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.932+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:50.948 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:50.948 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:50.948 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:50.948 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:50.948 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:50.949 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:50.949 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:50.949 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:50.949 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:50.950 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:50.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0054040 con 0x7f63e0064730 2021-03-09T08:57:50.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.934+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:50.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.934+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:57:50.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:50.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T08:57:51.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:51.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:51.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:51.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:52.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:52.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:52.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:52.410+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:53.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:53.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:53.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:53.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:54.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:54.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:54.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:54.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:54.448+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:57:55.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:55.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e006a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:57:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e006a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:57:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006a9b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f63f0065630 tx=0x7f63f0054250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:57:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006a9b0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f63f0065630 tx=0x7f63f0054250).stop 2021-03-09T08:57:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:57:55.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e006a9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:57:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e006a9b0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:57:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:57:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:57:55.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:57:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:57:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:57:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:57:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d6a0 con 0x7f6400145bd0 2021-03-09T08:57:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:57:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012e40 con 0x7f6400146050 2021-03-09T08:57:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.927+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:57:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:57:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:57:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:57:55.935 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3934 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:57:55.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:57:55.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:57:55.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:57:55.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:57:55.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:57:55.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:57:55.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.928+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:57:55.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:57:55.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:55.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:55.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:57:55.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:57:55.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:57:55.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:57:55.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:57:55.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e320 con 0x7f6400146050 2021-03-09T08:57:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:57:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:57:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015370 con 0x7f6400146050 2021-03-09T08:57:55.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:57:55.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:55.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63f005c330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:57:55.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:57:55.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.929+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:57:55.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.931+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T08:57:55.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.931+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:57:55.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.931+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:57:55.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.931+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:57:55.944 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:57:55.944 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:57:55.945 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:55.945 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:57:55.945 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:57:55.945 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:57:55.945 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:57:55.946 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:57:55.946 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:57:55.946 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:57:55.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0054040 con 0x7f6400146050 2021-03-09T08:57:55.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.933+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:57:55.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T08:57:55.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:55.933+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:57:56.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:56.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:56.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:56.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:57.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:57.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:57.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:57.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:58.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:58.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:58.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:58.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:59.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:59.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:57:59.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:59.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:57:59.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:57:59.449+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:00.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:00.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:00.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63f005c330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63f005c330).stop 2021-03-09T08:58:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:58:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00783f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:00.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:00.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:00.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:00.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:58:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c480 con 0x7f6400145bd0 2021-03-09T08:58:00.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d690 con 0x7f63e0064730 2021-03-09T08:58:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:00.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:00.936 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 3966 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:00.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00783f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:58:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:00.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:00.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:00.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:00.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.929+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:00.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:00.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:58:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:58:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:00.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065cf0 con 0x7f63e0064730 2021-03-09T08:58:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006c5b0 con 0x7f63e0064730 2021-03-09T08:58:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:58:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00783f0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4046440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:58:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.930+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c040 con 0x7f63e0064730 2021-03-09T08:58:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.932+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.932+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.932+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:00.951 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63e0064730 2021-03-09T08:58:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.934+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.934+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:58:00.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:00.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T08:58:01.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:01.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:01.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:01.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:02.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:02.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:02.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:02.411+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:03.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:03.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:03.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:03.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:04.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:04.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:04.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:04.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:04.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:04.450+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:05.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:05.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:05.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00783f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:58:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00783f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00783f0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4046440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00783f0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4046440).stop 2021-03-09T08:58:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00783f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00783f0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T08:58:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017810 con 0x7f63e00070a0 2021-03-09T08:58:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e530 con 0x7f6400145bd0 2021-03-09T08:58:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:58:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:05.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:05.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:05.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:05.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.929+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:05.937 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:05.930+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4003 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:05.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.930+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:05.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.930+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:58:05.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.930+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:05.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.930+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:05.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.931+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:05.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.931+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:05.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.931+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:05.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:05.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:05.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:05.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:05.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:05.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:58:05.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:58:05.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:05.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:05.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000bd40 con 0x7f6400146050 2021-03-09T08:58:05.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:05.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:05.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e320 con 0x7f6400146050 2021-03-09T08:58:05.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:58:05.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.932+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:05.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.933+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:58:05.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.933+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:05.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f63e0011a90 tx=0x7f63f0013f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:05.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00542c0 con 0x7f6400146050 2021-03-09T08:58:05.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.933+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:05.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.933+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:05.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.933+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:05.947 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:05.948 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:05.948 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:05.948 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:05.948 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:05.949 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:05.949 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:05.949 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:05.949 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:05.950 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:05.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00641a0 con 0x7f6400146050 2021-03-09T08:58:05.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.935+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:05.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.935+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:58:05.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:05.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T08:58:06.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:06.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:06.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:06.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:07.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:07.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:07.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:07.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:08.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:08.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:08.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:08.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:09.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:09.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:09.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:09.451+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:10.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:58:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f63e0011a90 tx=0x7f63f0013f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 rx=0x7f63e0011a90 tx=0x7f63f0013f00).stop 2021-03-09T08:58:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:58:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c470 con 0x7f63e00070a0 2021-03-09T08:58:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d6a0 con 0x7f6400145bd0 2021-03-09T08:58:10.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:10.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:58:10.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.930+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:10.938 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:10.931+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4028 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:10.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.931+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:10.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.931+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:10.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:10.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:10.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.931+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:10.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.931+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:58:10.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.931+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:10.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.931+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:10.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:10.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:10.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:10.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:10.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:10.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:10.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:10.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:10.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:58:10.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:58:10.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:10.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:10.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T08:58:10.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:10.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:10.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001ae20 con 0x7f63e0064730 2021-03-09T08:58:10.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:58:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.932+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.933+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.933+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:58:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.933+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T08:58:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.934+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.934+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.934+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:10.949 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:10.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047cc0 con 0x7f63e0064730 2021-03-09T08:58:10.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.935+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:10.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.935+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:58:10.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:10.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01cc40 con 0x7f63e0064730 2021-03-09T08:58:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:11.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:11.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:12.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:12.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:12.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:12.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:12.412+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:13.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:13.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:13.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:13.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:14.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:14.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:14.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:14.452+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:15.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:15.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:15.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:15.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:15.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).stop 2021-03-09T08:58:15.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:15.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:15.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:58:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:15.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:15.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022740 con 0x7f63e00070a0 2021-03-09T08:58:15.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:15.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023c70 con 0x7f6400145bd0 2021-03-09T08:58:15.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:15.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e490 con 0x7f6400146050 2021-03-09T08:58:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.930+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:15.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:15.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:58:15.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:15.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:15.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:15.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:15.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:15.941 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4076 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:15.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:15.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:15.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:15.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:15.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:15.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:58:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:58:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006c930 con 0x7f6400146050 2021-03-09T08:58:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000bd40 con 0x7f6400146050 2021-03-09T08:58:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:58:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.931+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.932+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f0054330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.932+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:58:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.932+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T08:58:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.934+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.934+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.934+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:15.949 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005c170 con 0x7f6400146050 2021-03-09T08:58:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.934+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.934+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:58:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:15.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T08:58:16.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:16.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:16.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:16.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:17.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:17.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:17.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:17.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:18.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:18.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:18.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:18.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:19.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:19.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:19.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:19.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:19.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:19.452+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:20.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:20.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:20.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:21.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:21.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f0054330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:21.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f0054330).stop 2021-03-09T08:58:21.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b080 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:58:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a220 con 0x7f63e00070a0 2021-03-09T08:58:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014450 con 0x7f6400145bd0 2021-03-09T08:58:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007aea0 con 0x7f63e0064730 2021-03-09T08:58:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.931+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:21.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:21.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:21.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:21.013 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4111 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:21.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:21.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:21.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:58:21.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:21.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:21.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:21.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.932+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:21.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:21.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:21.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:21.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:21.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:21.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:58:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:58:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015370 con 0x7f63e0064730 2021-03-09T08:58:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T08:58:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:58:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.933+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.934+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:58:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.934+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.934+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc046b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00f350 con 0x7f63e0064730 2021-03-09T08:58:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.936+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:21.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.936+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:21.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.936+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:21.021 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:21.021 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:21.021 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:21.023 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:21.023 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:21.023 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:21.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00f350 con 0x7f63e0064730 2021-03-09T08:58:21.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.936+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:21.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.936+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:58:21.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:20.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T08:58:21.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:21.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:21.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:22.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:22.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:22.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:22.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:22.412+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:23.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:23.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:23.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:23.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:24.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:24.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:24.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:24.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:24.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:24.453+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:25.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:25.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:25.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:25.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:25.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc046b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:25.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc046b50).stop 2021-03-09T08:58:25.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:25.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:25.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:25.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:25.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:58:25.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00783f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:25.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:25.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:25.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:58:25.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:25.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e10 con 0x7f6400145bd0 2021-03-09T08:58:25.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:25.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022740 con 0x7f6400146050 2021-03-09T08:58:25.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.932+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:25.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.933+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.933+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:25.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.933+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00783f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.933+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:58:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.933+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:25.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.933+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:25.943 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:25.933+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4153 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:25.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:25.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:25.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:25.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:25.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:25.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:25.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:25.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:25.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:58:25.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:58:25.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:25.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:25.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00101f0 con 0x7f6400146050 2021-03-09T08:58:25.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:25.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:25.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065cf0 con 0x7f6400146050 2021-03-09T08:58:25.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:58:25.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.934+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:25.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.935+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:58:25.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00783f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f4022080).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:25.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.935+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:25.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c040 con 0x7f6400146050 2021-03-09T08:58:25.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.936+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:25.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.936+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:25.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.936+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:25.951 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:25.952 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:25.952 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:25.952 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:25.952 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:25.952 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:25.953 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:25.953 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:25.953 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:25.953 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:25.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T08:58:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.937+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.937+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:58:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:25.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:58:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:26.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:26.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:27.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:27.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:27.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:27.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:28.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:28.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:28.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:28.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:29.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:29.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:29.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:29.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:29.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:29.454+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:30.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:30.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:30.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00783f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:30.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00783f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:30.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00783f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f4022080).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00783f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f4022080).stop 2021-03-09T08:58:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:30.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:30.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00783f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:30.938 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4179 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:30.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00783f0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:30.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:58:30.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:30.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.933+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:30.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:30.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:30.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00183f0 con 0x7f63e00070a0 2021-03-09T08:58:30.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:30.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f6400145bd0 2021-03-09T08:58:30.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:30.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00024f0 con 0x7f63e0064730 2021-03-09T08:58:30.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:30.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:30.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:58:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:30.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:30.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.934+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:30.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:30.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:30.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:30.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:30.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:30.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:58:30.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:58:30.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:30.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:30.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029700 con 0x7f63e0064730 2021-03-09T08:58:30.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:30.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:30.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015370 con 0x7f63e0064730 2021-03-09T08:58:30.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:58:30.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:30.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:58:30.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029210 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:30.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.935+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:30.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04bc90 con 0x7f63e0064730 2021-03-09T08:58:30.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.936+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:30.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.936+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:30.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.936+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:30.951 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:30.952 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:30.952 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:30.952 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:30.954 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:30.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04bc90 con 0x7f63e0064730 2021-03-09T08:58:30.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.936+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:30.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.936+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:58:30.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:30.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T08:58:31.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:31.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:31.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:31.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:32.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:32.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:32.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:32.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:32.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:32.414+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:33.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:33.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:33.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:33.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:34.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:34.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:34.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:34.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:34.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:34.455+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:35.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:35.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:35.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0029210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:58:35.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0029210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:35.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029210 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:35.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029210 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009a00).stop 2021-03-09T08:58:35.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:35.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:35.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0029210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:35.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029210 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:35.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T08:58:35.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:35.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:35.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:35.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:58:35.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:35.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00153c0 con 0x7f6400145bd0 2021-03-09T08:58:35.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:35.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017810 con 0x7f6400146050 2021-03-09T08:58:35.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.934+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:35.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:35.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:35.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:35.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:35.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:35.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:58:35.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:35.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:35.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:35.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:35.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:35.945 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:35.935+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4218 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:35.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:35.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:35.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:35.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:35.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:35.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T08:58:35.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T08:58:35.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:35.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:35.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00226f0 con 0x7f6400146050 2021-03-09T08:58:35.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:35.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:35.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006c930 con 0x7f6400146050 2021-03-09T08:58:35.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T08:58:35.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:35.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078ca0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7f63e006b430 tx=0x7f63f0006e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:35.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:58:35.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.936+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:35.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T08:58:35.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.938+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:35.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.938+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:35.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.938+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:35.952 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:35.952 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:35.952 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:35.952 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:35.954 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:35.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000d280 con 0x7f6400146050 2021-03-09T08:58:35.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.938+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:35.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.938+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:58:35.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:35.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T08:58:36.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:36.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:36.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:36.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:37.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:37.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:37.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:37.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:38.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:38.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:39.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:39.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:39.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:39.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:39.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:39.456+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:40.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:40.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:40.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0078ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:58:40.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0078ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:40.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078ca0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7f63e006b430 tx=0x7f63f0006e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:40.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078ca0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 rx=0x7f63e006b430 tx=0x7f63f0006e00).stop 2021-03-09T08:58:40.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:40.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:40.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0078ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:40.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078ca0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:40.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:58:40.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:40.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:40.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:40.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:40.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:40.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:40.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:40.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ad0 con 0x7f63e00070a0 2021-03-09T08:58:40.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:40.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014450 con 0x7f6400145bd0 2021-03-09T08:58:40.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:40.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f9f0 con 0x7f63e0064730 2021-03-09T08:58:40.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.935+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:40.943 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4251 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:40.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:40.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:40.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:40.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:58:40.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:40.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:40.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:40.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:40.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:40.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:40.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.936+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:40.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:40.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:40.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:40.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:40.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:40.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:58:40.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:58:40.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:40.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:40.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024a50 con 0x7f63e0064730 2021-03-09T08:58:40.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:40.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:40.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00101f0 con 0x7f63e0064730 2021-03-09T08:58:40.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:58:40.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.937+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.938+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:58:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.938+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f4000e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.938+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c040 con 0x7f63e0064730 2021-03-09T08:58:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.939+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.939+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.939+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:40.954 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:40.954 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:40.954 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:40.954 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:40.955 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40098b0 con 0x7f63e0064730 2021-03-09T08:58:40.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.943+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:40.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.943+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:58:40.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:40.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T08:58:41.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:41.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:41.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:41.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:42.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:42.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:42.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:42.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:42.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:42.415+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:43.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:43.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:43.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:43.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:44.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:44.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:44.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:44.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:44.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:44.457+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:45.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:45.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:58:45.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:45.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f4000e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:45.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f4000e00).stop 2021-03-09T08:58:45.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:45.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:45.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:45.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:45.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:58:45.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:45.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:45.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.936+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00162b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:45.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:45.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:45.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:45.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:45.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c680 con 0x7f63e00070a0 2021-03-09T08:58:45.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:45.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00155c0 con 0x7f6400145bd0 2021-03-09T08:58:45.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:45.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:58:45.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:45.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:45.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:45.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:45.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:45.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:45.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:58:45.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:45.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:45.946 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:45.937+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4305 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:45.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.938+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:45.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.938+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:45.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.938+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:45.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.938+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:45.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.938+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:45.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.938+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:45.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.938+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:45.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.938+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:45.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:58:45.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:58:45.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:45.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:45.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00167a0 con 0x7f6400146050 2021-03-09T08:58:45.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:45.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:45.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029700 con 0x7f6400146050 2021-03-09T08:58:45.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:58:45.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.939+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:45.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.940+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:58:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.940+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00162b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc047130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.940+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.941+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0715c0 con 0x7f6400146050 2021-03-09T08:58:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.941+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:45.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.941+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:45.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.941+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:45.956 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:45.956 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:45.956 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:45.956 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:45.956 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:45.957 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:45.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T08:58:45.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.943+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:45.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.943+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:58:45.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:45.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f6400146050 2021-03-09T08:58:46.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:46.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:46.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:46.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:47.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:47.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:48.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:48.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:48.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:48.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:49.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:49.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:49.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:49.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:49.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:49.458+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:50.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:50.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:50.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.937+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:58:50.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.937+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00162b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:50.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.937+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00162b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc047130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:50.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.937+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00162b0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc047130).stop 2021-03-09T08:58:50.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:50.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:50.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00162b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:50.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00162b0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:50.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:58:50.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:50.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:50.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:50.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:50.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c470 con 0x7f63e00070a0 2021-03-09T08:58:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:50.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f340 con 0x7f6400145bd0 2021-03-09T08:58:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078630 con 0x7f63e0064730 2021-03-09T08:58:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:50.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:50.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:50.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:50.946 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:50.938+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4328 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:50.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.939+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:50.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.939+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:58:50.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.939+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:50.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.939+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:50.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.939+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:50.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.939+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:50.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.939+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:50.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:50.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:50.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:50.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:50.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:50.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:58:50.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:58:50.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:50.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:50.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00213d0 con 0x7f63e0064730 2021-03-09T08:58:50.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:50.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:50.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024a50 con 0x7f63e0064730 2021-03-09T08:58:50.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:58:50.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.940+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:50.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.942+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f63f4070560 tx=0x7f63f40038a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:50.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.942+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:58:50.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.942+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:50.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c040 con 0x7f63e0064730 2021-03-09T08:58:50.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.944+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:50.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.944+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:50.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.944+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:50.954 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:50.956 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:50.956 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:50.956 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:50.956 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:50.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40098b0 con 0x7f63e0064730 2021-03-09T08:58:50.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.946+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:50.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.946+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:58:50.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:50.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T08:58:51.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:51.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:51.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:51.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:52.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:52.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:52.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:52.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:52.416+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:53.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:53.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:53.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:53.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:54.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:54.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:54.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:54.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:54.459+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:58:55.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:55.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.938+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:58:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.938+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:58:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.938+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f63f4070560 tx=0x7f63f40038a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:58:55.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.938+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 rx=0x7f63f4070560 tx=0x7f63f40038a0).stop 2021-03-09T08:58:55.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:55.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:58:55.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:58:55.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:58:55.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T08:58:55.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:55.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:58:55.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:55.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:58:55.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:58:55.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:58:55.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:55.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015fc0 con 0x7f63e00070a0 2021-03-09T08:58:55.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:55.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T08:58:55.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:58:55.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f340 con 0x7f6400146050 2021-03-09T08:58:55.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:58:55.947 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4376 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:58:55.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:58:55.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.939+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:55.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:58:55.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:58:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:58:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:58:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:58:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:58:55.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:58:55.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:58:55.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:58:55.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:58:55.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:58:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:58:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:58:55.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:58:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:58:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00785e0 con 0x7f6400146050 2021-03-09T08:58:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:58:55.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:58:55.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00167a0 con 0x7f6400146050 2021-03-09T08:58:55.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:58:55.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.940+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:55.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.942+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b450 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f63fc0335d0 tx=0x7f63fc0712f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:58:55.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.942+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:58:55.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.942+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:58:55.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc033640 con 0x7f6400146050 2021-03-09T08:58:55.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.942+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:58:55.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.942+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:58:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.942+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:58:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:58:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:58:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:58:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:58:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:58:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:58:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:58:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:58:55.958 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:58:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc033640 con 0x7f6400146050 2021-03-09T08:58:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.943+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:58:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.943+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:58:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:55.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f6400146050 2021-03-09T08:58:56.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:56.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:56.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:56.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:57.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:57.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:57.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:57.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:58.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:58.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:58.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:58.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:59.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:59.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:58:59.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:59.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:58:59.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:58:59.460+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:00.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:00.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b450 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f63fc0335d0 tx=0x7f63fc0712f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b450 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 rx=0x7f63fc0335d0 tx=0x7f63fc0712f0).stop 2021-03-09T08:59:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b450 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:59:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001aa90 con 0x7f63e00070a0 2021-03-09T08:59:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028c00 con 0x7f6400145bd0 2021-03-09T08:59:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:00.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014450 con 0x7f63e0064730 2021-03-09T08:59:00.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:00.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:00.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:00.948 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4411 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:00.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.940+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.941+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:59:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.941+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.941+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.941+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.941+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.941+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:00.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:00.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:00.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:00.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:00.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:59:00.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:59:00.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:00.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:00.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T08:59:00.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:00.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:00.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00213d0 con 0x7f63e0064730 2021-03-09T08:59:00.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:59:00.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.942+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:00.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.944+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7f63f40187c0 tx=0x7f63f4018910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:00.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.944+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:59:00.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.944+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:00.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f63e0064730 2021-03-09T08:59:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.946+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.946+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.946+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:00.959 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:00.959 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:00.959 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:00.959 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:00.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.948+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4018590 con 0x7f63e0064730 2021-03-09T08:59:00.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.948+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:00.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.948+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:59:00.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:00.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f63e0064730 2021-03-09T08:59:01.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:01.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:01.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:01.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:02.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:02.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:02.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:02.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:02.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:02.417+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:03.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:03.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:03.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:03.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:04.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:04.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:04.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:04.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:04.461+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:05.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:05.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:05.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:59:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7f63f40187c0 tx=0x7f63f4018910).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 rx=0x7f63f40187c0 tx=0x7f63f4018910).stop 2021-03-09T08:59:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:59:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:06.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:06.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f9f0 con 0x7f63e00070a0 2021-03-09T08:59:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079670 con 0x7f6400145bd0 2021-03-09T08:59:06.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:06.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T08:59:06.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:06.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:06.038 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:05.941+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4450 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.942+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.942+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:59:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.942+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.942+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.942+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.942+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.942+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:59:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:59:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0020fd0 con 0x7f6400146050 2021-03-09T08:59:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T08:59:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:59:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.944+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.946+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013660 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 rx=0x7f63f40348e0 tx=0x7f63f4034a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.947+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:59:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.947+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c040 con 0x7f6400146050 2021-03-09T08:59:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.947+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.947+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.947+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:06.056 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034e90 con 0x7f6400146050 2021-03-09T08:59:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.949+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.949+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:59:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:05.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4023d70 con 0x7f6400146050 2021-03-09T08:59:06.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:06.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:06.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:06.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:07.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:07.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:07.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:07.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:08.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:08.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:08.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:08.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:09.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:09.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:09.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:09.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:09.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:09.462+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:10.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:10.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:10.944 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:10.942+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4481 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:10.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.942+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0013660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T08:59:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.942+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0013660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.942+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013660 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 rx=0x7f63f40348e0 tx=0x7f63f4034a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.942+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013660 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 rx=0x7f63f40348e0 tx=0x7f63f4034a30).stop 2021-03-09T08:59:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013660 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:59:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:10.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006db70 con 0x7f63e00070a0 2021-03-09T08:59:10.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:10.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f6400145bd0 2021-03-09T08:59:10.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:10.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0064730 2021-03-09T08:59:10.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:10.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:10.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:10.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:10.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.943+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:59:10.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:10.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:10.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:10.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:10.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:10.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:10.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:10.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:10.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:10.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:59:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:59:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T08:59:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00785e0 con 0x7f63e0064730 2021-03-09T08:59:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:59:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.944+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.945+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bc10 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.945+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:59:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.945+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:10.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc032390 con 0x7f63e0064730 2021-03-09T08:59:10.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.945+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:10.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.945+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:10.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.945+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:10.963 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:10.963 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:10.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032390 con 0x7f63e0064730 2021-03-09T08:59:10.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.947+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:10.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.947+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:59:10.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:10.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T08:59:11.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:11.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:11.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:11.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:12.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:12.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:12.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:12.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:12.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:12.419+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:13.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:13.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:13.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:13.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:14.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:14.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:14.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:14.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:14.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:14.463+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:15.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:15.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bc10 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bc10 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc009a70).stop 2021-03-09T08:59:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002bc10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bc10 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:59:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00185f0 con 0x7f63e00070a0 2021-03-09T08:59:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f6400145bd0 2021-03-09T08:59:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00188e0 con 0x7f6400146050 2021-03-09T08:59:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:15.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:15.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:15.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:15.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:15.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:15.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:59:15.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:15.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:15.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:15.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:15.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.943+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:15.953 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4519 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:15.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:15.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:15.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:15.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:15.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:15.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:59:15.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:59:15.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:15.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:15.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00785e0 con 0x7f6400146050 2021-03-09T08:59:15.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:15.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:15.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0020fd0 con 0x7f6400146050 2021-03-09T08:59:15.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:59:15.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.944+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:15.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:59:15.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:15.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002f0f0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f40461f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:15.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c070 con 0x7f6400146050 2021-03-09T08:59:15.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:15.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:15.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:15.960 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:15.960 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:15.960 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:15.960 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:15.960 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:15.961 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:15.961 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:15.961 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:15.961 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:15.962 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:15.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4023280 con 0x7f6400146050 2021-03-09T08:59:15.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:15.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.945+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:59:15.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:15.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:59:16.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:16.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:16.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:16.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:17.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:17.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:17.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:17.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:18.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:18.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:18.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:18.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:19.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:19.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:19.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:19.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:19.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:19.464+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:20.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:20.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:20.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:59:20.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:20.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002f0f0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f40461f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:20.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002f0f0 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f40461f0).stop 2021-03-09T08:59:20.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:20.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:20.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002f0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002f0f0 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T08:59:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T08:59:20.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:20.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400145bd0 2021-03-09T08:59:20.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:20.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f9f0 con 0x7f63e0064730 2021-03-09T08:59:20.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:20.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:20.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:20.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:20.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.944+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:20.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.945+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:20.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.945+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:59:20.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.945+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:20.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.945+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:20.954 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:20.945+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4562 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:20.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.946+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:20.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.946+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.946+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:20.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:20.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:59:20.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:59:20.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:20.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:20.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028bb0 con 0x7f63e0064730 2021-03-09T08:59:20.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:20.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:20.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00785e0 con 0x7f63e0064730 2021-03-09T08:59:20.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:59:20.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.947+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:20.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.948+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 rx=0x7f63f40485d0 tx=0x7f63f4052300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:20.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.949+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:59:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.949+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c040 con 0x7f63e0064730 2021-03-09T08:59:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.951+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.951+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.951+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:20.964 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4018400 con 0x7f63e0064730 2021-03-09T08:59:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.953+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.953+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:59:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:20.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4070170 con 0x7f63e0064730 2021-03-09T08:59:21.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:21.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:21.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:21.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:22.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:22.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:22.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:22.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:22.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:22.420+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:23.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:23.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:23.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:23.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:24.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:24.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:24.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:24.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:24.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:24.465+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:25.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:25.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:25.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:25.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:25.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 rx=0x7f63f40485d0 tx=0x7f63f4052300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:25.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 rx=0x7f63f40485d0 tx=0x7f63f4052300).stop 2021-03-09T08:59:25.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:25.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:25.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:25.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:25.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:59:25.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:25.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:25.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:25.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:25.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:25.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:25.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:25.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023e70 con 0x7f63e00070a0 2021-03-09T08:59:25.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:25.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00188e0 con 0x7f6400145bd0 2021-03-09T08:59:25.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:25.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00185f0 con 0x7f6400146050 2021-03-09T08:59:25.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:25.955 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4596 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:25.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:25.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:59:25.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.946+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.947+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.947+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.947+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:25.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:25.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:25.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:25.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:25.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:59:25.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:59:25.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:25.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:25.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00785e0 con 0x7f6400146050 2021-03-09T08:59:25.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:25.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:25.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028bb0 con 0x7f6400146050 2021-03-09T08:59:25.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:59:25.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.948+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:25.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.949+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012130 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7f63f40483a0 tx=0x7f63f4034be0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:25.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.949+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:59:25.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.949+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:25.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4051180 con 0x7f6400146050 2021-03-09T08:59:25.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.949+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:25.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.949+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:25.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.949+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:25.962 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:25.963 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:25.963 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:25.963 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:25.963 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:25.965 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4051180 con 0x7f6400146050 2021-03-09T08:59:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.955+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f6400146050 2021-03-09T08:59:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:25.956+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:59:26.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:26.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:26.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:26.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:27.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:27.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:27.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:27.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:28.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:28.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:28.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:28.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:29.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:29.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:29.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:29.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:29.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:29.466+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:30.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:30.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:30.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:30.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:30.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012130 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7f63f40483a0 tx=0x7f63f4034be0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:30.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012130 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 rx=0x7f63f40483a0 tx=0x7f63f4034be0).stop 2021-03-09T08:59:30.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:30.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:30.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:30.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012130 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:30.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:59:30.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:30.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:30.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:30.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63e00070a0 2021-03-09T08:59:30.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:30.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006cb80 con 0x7f6400145bd0 2021-03-09T08:59:30.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:30.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:59:30.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:30.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:30.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:30.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:30.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:30.956 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:30.947+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4636 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:30.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.948+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:30.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.948+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:59:30.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.948+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:30.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.948+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:30.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.949+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:30.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.949+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:30.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.949+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:30.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.950+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:30.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:30.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:30.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:30.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:30.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:59:30.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:59:30.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:30.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:30.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001dc0 con 0x7f63e0064730 2021-03-09T08:59:30.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:30.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:30.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T08:59:30.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:59:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:59:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024cf0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a7b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.952+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T08:59:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.952+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.952+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.952+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:30.966 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0715c0 con 0x7f63e0064730 2021-03-09T08:59:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.953+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.953+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:59:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:30.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T08:59:31.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:31.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:31.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:31.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:32.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:32.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:32.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:32.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:32.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:32.422+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:33.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:33.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:33.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:33.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:34.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:34.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:34.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:34.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:34.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:34.467+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:35.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:35.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:35.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0024cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:35.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0024cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:35.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024cf0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a7b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:35.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024cf0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a7b0).stop 2021-03-09T08:59:35.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:35.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:35.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0024cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:35.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024cf0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:35.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:59:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:35.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:35.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:35.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:35.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bd30 con 0x7f63e00070a0 2021-03-09T08:59:35.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:35.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00140c0 con 0x7f6400145bd0 2021-03-09T08:59:35.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:35.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400146050 2021-03-09T08:59:35.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:35.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:35.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:35.956 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:35.948+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4674 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:35.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:35.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:35.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:35.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:59:35.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:35.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:35.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:35.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:35.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:35.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:35.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:35.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:35.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:35.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:35.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:59:35.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:59:35.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:35.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:35.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016ce0 con 0x7f6400146050 2021-03-09T08:59:35.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:35.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:35.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00785e0 con 0x7f6400146050 2021-03-09T08:59:35.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:59:35.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.949+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:35.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027d90 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f40038a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:35.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:59:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402b130 con 0x7f6400146050 2021-03-09T08:59:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:35.967 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:35.967 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:35.967 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:35.967 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:35.967 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:35.968 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:35.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f402b130 con 0x7f6400146050 2021-03-09T08:59:35.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:35.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.950+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:59:35.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:35.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T08:59:36.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:36.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:36.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:36.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:37.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:37.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:37.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:37.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:38.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:38.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:38.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:38.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:39.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:39.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:39.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:39.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:39.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:39.468+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:40.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:40.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:40.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027d90 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f40038a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027d90 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f40038a0).stop 2021-03-09T08:59:40.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027d90 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:59:40.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:40.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:40.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:40.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:40.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00185f0 con 0x7f63e00070a0 2021-03-09T08:59:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:40.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00188e0 con 0x7f6400145bd0 2021-03-09T08:59:40.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:40.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T08:59:40.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:40.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:40.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:40.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:40.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:40.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:40.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:59:40.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:40.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:40.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:40.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:40.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:40.960 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:40.949+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4713 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:40.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:40.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:40.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:40.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:40.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:40.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:59:40.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:59:40.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:40.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:40.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014400 con 0x7f63e0064730 2021-03-09T08:59:40.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:40.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:40.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001dc0 con 0x7f63e0064730 2021-03-09T08:59:40.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:59:40.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:40.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:59:40.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f63fc071360 tx=0x7f63fc023f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:40.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.950+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:40.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.952+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e0064730 2021-03-09T08:59:40.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.952+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:40.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.952+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:40.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.952+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:40.967 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:40.967 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:40.967 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:40.967 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:40.967 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:40.968 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:40.968 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:40.968 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:40.969 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:40.969 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:40.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.952+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d7d0 con 0x7f63e0064730 2021-03-09T08:59:40.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.952+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:40.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.952+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:59:40.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:40.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63e0064730 2021-03-09T08:59:41.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:41.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:41.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:41.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:42.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:42.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:42.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:42.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:42.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:42.423+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:43.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:43.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:43.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:43.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:44.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:44.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:44.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:44.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:44.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:44.469+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:45.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:45.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f63fc071360 tx=0x7f63fc023f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:45.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 rx=0x7f63fc071360 tx=0x7f63fc023f10).stop 2021-03-09T08:59:45.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:45.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:45.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:45.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d810 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T08:59:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:45.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:45.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:45.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:45.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:45.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f340 con 0x7f63e00070a0 2021-03-09T08:59:45.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:45.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400145bd0 2021-03-09T08:59:45.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:45.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00230c0 con 0x7f6400146050 2021-03-09T08:59:45.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:45.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:45.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:45.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:45.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.950+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:45.959 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4753 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:45.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:45.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:59:45.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:45.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:45.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:45.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:45.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:45.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:45.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:45.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:45.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:45.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:45.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T08:59:45.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T08:59:45.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:45.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:45.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00785e0 con 0x7f6400146050 2021-03-09T08:59:45.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:45.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:45.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014400 con 0x7f6400146050 2021-03-09T08:59:45.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.952+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T08:59:45.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.952+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:45.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.952+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c930 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc051f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:45.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.952+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:59:45.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.952+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:45.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04f210 con 0x7f6400146050 2021-03-09T08:59:45.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.953+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:45.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.953+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:45.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.953+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:45.967 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:45.967 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:45.969 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04f3c0 con 0x7f6400146050 2021-03-09T08:59:45.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.953+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:45.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.953+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:59:45.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:45.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04fa90 con 0x7f6400146050 2021-03-09T08:59:46.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:46.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:46.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:46.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:47.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:47.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:47.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:47.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:48.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:48.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:48.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:48.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:49.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:49.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:49.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:49.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:49.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:49.470+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:50.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:50.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:50.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T08:59:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c930 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc051f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c930 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc051f20).stop 2021-03-09T08:59:50.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:50.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:50.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:50.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c930 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:50.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:59:50.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:50.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:50.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:50.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:50.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:50.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:50.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:50.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bd30 con 0x7f63e00070a0 2021-03-09T08:59:50.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:50.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a460 con 0x7f6400145bd0 2021-03-09T08:59:50.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:50.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022940 con 0x7f63e0064730 2021-03-09T08:59:50.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:50.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:50.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:50.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:50.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T08:59:50.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:50.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.951+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:50.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.952+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:50.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.952+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:50.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.952+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:50.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.952+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:50.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.952+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:50.962 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:50.952+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4784 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:50.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:50.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:50.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:50.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:50.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:50.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:59:50.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:59:50.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:50.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:50.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016b50 con 0x7f63e0064730 2021-03-09T08:59:50.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:50.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:50.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016ce0 con 0x7f63e0064730 2021-03-09T08:59:50.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:59:50.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:50.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f4034920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:50.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T08:59:50.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.953+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:50.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034c50 con 0x7f63e0064730 2021-03-09T08:59:50.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.954+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.954+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.954+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:50.971 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:50.971 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:50.972 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:50.972 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:50.972 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:50.972 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:50.972 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:50.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4006a90 con 0x7f63e0064730 2021-03-09T08:59:50.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.955+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:50.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.955+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T08:59:50.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:50.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T08:59:51.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:51.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:51.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:52.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:52.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:52.424+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:53.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:53.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:53.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:53.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:54.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:54.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:54.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:54.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:54.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:54.471+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T08:59:55.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:55.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T08:59:55.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T08:59:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f4034920).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T08:59:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f4034920).stop 2021-03-09T08:59:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:55.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T08:59:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000dfd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T08:59:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T08:59:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T08:59:55.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T08:59:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T08:59:55.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T08:59:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T08:59:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00188e0 con 0x7f63e00070a0 2021-03-09T08:59:55.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f6400145bd0 2021-03-09T08:59:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T08:59:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000da50 con 0x7f6400146050 2021-03-09T08:59:55.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T08:59:55.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T08:59:55.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:55.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T08:59:55.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T08:59:55.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T08:59:55.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T08:59:55.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T08:59:55.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T08:59:55.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T08:59:55.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T08:59:55.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T08:59:55.964 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T08:59:55.953+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4819 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T08:59:55.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T08:59:55.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:55.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:55.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T08:59:55.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T08:59:55.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T08:59:55.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T08:59:55.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T08:59:55.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:55.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f800 con 0x7f6400146050 2021-03-09T08:59:55.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T08:59:55.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T08:59:55.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016b50 con 0x7f6400146050 2021-03-09T08:59:55.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T08:59:55.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:55.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001d320 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7f63f4061360 tx=0x7f63f4016fc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T08:59:55.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T08:59:55.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.954+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T08:59:55.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40611a0 con 0x7f6400146050 2021-03-09T08:59:55.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.955+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T08:59:55.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.955+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T08:59:55.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.955+0000 7f63edffb700 10 monclient: dump: 2021-03-09T08:59:55.971 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T08:59:55.971 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T08:59:55.971 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:55.971 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T08:59:55.972 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T08:59:55.972 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T08:59:55.972 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T08:59:55.972 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T08:59:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T08:59:55.973 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T08:59:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4061600 con 0x7f6400146050 2021-03-09T08:59:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.955+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T08:59:55.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.955+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T08:59:55.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:55.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T08:59:56.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:56.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:56.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:56.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:57.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:57.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:57.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:57.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:58.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:58.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:58.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:58.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:59.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T08:59:59.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:59.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T08:59:59.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T08:59:59.472+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:00.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:00.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001d320 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7f63f4061360 tx=0x7f63f4016fc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001d320 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 rx=0x7f63f4061360 tx=0x7f63f4016fc0).stop 2021-03-09T09:00:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001d320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001d320 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:00.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:00:00.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:00.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:00.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:00.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:00.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:00.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:00.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:00.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f9f0 con 0x7f63e00070a0 2021-03-09T09:00:00.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:00.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bd30 con 0x7f6400145bd0 2021-03-09T09:00:00.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:00.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029cd0 con 0x7f63e0064730 2021-03-09T09:00:00.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:00.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:00.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:00.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:00.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:00.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:00.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:00:00.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:00.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:00.965 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4854 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:00.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:00.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:00.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.954+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:00.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:00.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:00.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:00.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:00.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:00.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:00:00.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:00:00.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:00.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:00.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f63e0064730 2021-03-09T09:00:00.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:00.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:00.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00226f0 con 0x7f63e0064730 2021-03-09T09:00:00.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:00:00.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.955+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:00.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f63e0010810 tx=0x7f63f0054250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:00.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.956+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:00:00.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.956+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:00.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.958+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0064730 2021-03-09T09:00:00.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.958+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:00.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.958+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:00.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.958+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:00.972 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:00.972 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:00.972 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:00.972 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:00.972 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:00.973 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:00.973 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:00.973 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:00.973 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:00.974 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.958+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0054040 con 0x7f63e0064730 2021-03-09T09:00:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.958+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.958+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:00:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:00.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T09:00:01.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:01.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:01.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:01.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:02.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:02.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:02.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:02.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:02.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:02.425+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:03.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:03.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:03.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:03.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:04.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:04.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:04.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:04.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:04.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:04.473+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:05.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:05.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:05.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:00:05.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:05.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f63e0010810 tx=0x7f63f0054250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:05.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 rx=0x7f63e0010810 tx=0x7f63f0054250).stop 2021-03-09T09:00:05.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:05.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:05.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000dfd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:05.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000dfd0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:05.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:00:05.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:05.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:05.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:05.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:05.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:05.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:05.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:05.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f340 con 0x7f63e00070a0 2021-03-09T09:00:05.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:05.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:00:05.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:05.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00188e0 con 0x7f6400146050 2021-03-09T09:00:05.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:05.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:05.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:05.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:05.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:05.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:05.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.956+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:00:05.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.956+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:05.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.956+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:05.965 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:05.955+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4893 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:05.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.956+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:05.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.956+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:05.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.956+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:05.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:05.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:05.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:05.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:05.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:05.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:00:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:00:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f6400146050 2021-03-09T09:00:05.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f6400146050 2021-03-09T09:00:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:00:05.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.957+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.958+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000eb40 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f002d5c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.958+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:00:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.958+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00157f0 con 0x7f6400146050 2021-03-09T09:00:05.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.961+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.961+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.961+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:05.972 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:05.974 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0026370 con 0x7f6400146050 2021-03-09T09:00:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.962+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.962+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:00:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:05.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T09:00:06.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:06.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:06.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:06.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:07.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:07.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:07.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:07.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:08.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:08.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:08.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:08.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:09.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:09.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:09.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:09.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:09.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:09.474+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:10.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:10.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:00:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:10.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000eb40 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f002d5c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:10.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000eb40 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f002d5c0).stop 2021-03-09T09:00:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000eb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000eb40 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:00:10.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:10.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:10.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:10.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:10.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:10.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:10.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:10.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027000 con 0x7f63e00070a0 2021-03-09T09:00:10.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:10.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f6400145bd0 2021-03-09T09:00:10.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:10.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013080 con 0x7f63e0064730 2021-03-09T09:00:10.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:10.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:10.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:10.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:10.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:10.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.957+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:10.969 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:10.957+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4931 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:10.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.957+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:00:10.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.957+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:10.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.957+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:10.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.957+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:10.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.957+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:10.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.957+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:10.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:10.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:10.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:10.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:10.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:10.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:00:10.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:00:10.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:10.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:10.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f180 con 0x7f63e0064730 2021-03-09T09:00:10.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:10.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:10.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00785e0 con 0x7f63e0064730 2021-03-09T09:00:10.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:00:10.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:10.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.958+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014ae0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:10.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.959+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:00:10.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.959+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:10.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f63e0064730 2021-03-09T09:00:10.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.960+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:10.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.960+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:10.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.960+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:10.986 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:10.987 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:10.987 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:10.988 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:10.988 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:10.988 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:10.989 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:10.990 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:10.990 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:10.991 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:10.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0064730 2021-03-09T09:00:10.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.960+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.960+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:00:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:10.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0641a0 con 0x7f63e0064730 2021-03-09T09:00:11.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:11.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:11.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:11.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:12.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:12.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:12.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:12.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:12.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:12.427+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:13.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:13.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:13.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:13.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:14.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:14.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:14.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:14.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:14.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:14.475+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:15.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:15.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:15.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:15.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:15.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014ae0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:15.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014ae0 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).stop 2021-03-09T09:00:15.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:15.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:15.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:15.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:15.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:00:15.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:15.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:15.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:15.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:15.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0079d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:15.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:15.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:15.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014810 con 0x7f63e00070a0 2021-03-09T09:00:15.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:15.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ac90 con 0x7f6400145bd0 2021-03-09T09:00:15.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:15.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00258e0 con 0x7f6400146050 2021-03-09T09:00:15.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:15.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:15.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:15.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:15.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:15.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0079d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:15.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:00:15.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:15.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:15.970 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 4983 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:15.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:15.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:15.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.958+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:15.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:15.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:15.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:15.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:15.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:15.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:00:15.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:00:15.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:15.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:15.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f670 con 0x7f6400146050 2021-03-09T09:00:15.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:15.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:15.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f6400146050 2021-03-09T09:00:15.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:00:15.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:15.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:00:15.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:15.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.959+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0079d80 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f63f0037ff0 tx=0x7f63f0064210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:15.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:00:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.963+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.963+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.963+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:15.979 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:15.979 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:15.979 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:15.979 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:15.980 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:15.980 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:15.980 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:15.980 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:15.980 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:15.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004dcf0 con 0x7f6400146050 2021-03-09T09:00:15.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.963+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:15.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.963+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:00:15.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:15.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f6400146050 2021-03-09T09:00:16.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:16.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:16.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:16.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:17.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:17.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:17.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:17.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:18.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:18.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:18.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:18.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:19.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:19.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:19.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:19.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:19.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:19.476+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:20.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:20.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0079d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:00:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0079d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0079d80 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f63f0037ff0 tx=0x7f63f0064210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0079d80 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 rx=0x7f63f0037ff0 tx=0x7f63f0064210).stop 2021-03-09T09:00:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0079d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0079d80 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:00:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:20.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:20.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:20.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63e00070a0 2021-03-09T09:00:20.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:20.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:00:20.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:20.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f63e0064730 2021-03-09T09:00:20.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:20.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:20.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:20.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:20.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:20.967 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:20.959+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5006 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:20.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:20.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.960+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:00:20.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.960+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:20.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.960+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:20.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.960+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:20.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.960+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:20.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.960+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:20.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:20.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:20.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:20.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:20.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:20.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:00:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:00:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018890 con 0x7f63e0064730 2021-03-09T09:00:20.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f800 con 0x7f63e0064730 2021-03-09T09:00:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:00:20.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.961+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.962+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000eb40 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f40461f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.962+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:00:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.962+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402ae20 con 0x7f63e0064730 2021-03-09T09:00:20.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.963+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.963+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.963+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:20.978 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f402ae20 con 0x7f63e0064730 2021-03-09T09:00:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.966+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.966+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:00:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:20.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T09:00:21.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:21.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:21.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:21.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:22.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:22.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:22.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:22.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:22.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:22.428+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:23.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:23.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:23.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:24.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:24.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:24.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:24.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:24.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:24.477+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:25.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:25.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000eb40 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f40461f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000eb40 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f40461f0).stop 2021-03-09T09:00:25.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:25.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:25.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000eb40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000eb40 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:00:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:25.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:25.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:25.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:25.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:25.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e660 con 0x7f63e00070a0 2021-03-09T09:00:25.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:25.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014810 con 0x7f6400145bd0 2021-03-09T09:00:25.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:25.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bd00 con 0x7f6400146050 2021-03-09T09:00:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:25.970 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5054 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:25.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:25.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:00:25.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:25.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:25.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:25.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:25.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:25.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:25.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:25.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:00:25.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:00:25.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:25.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:25.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006db20 con 0x7f6400146050 2021-03-09T09:00:25.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:25.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:25.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f670 con 0x7f6400146050 2021-03-09T09:00:25.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:00:25.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.961+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:25.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.962+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bb20 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 rx=0x7f63e0025dc0 tx=0x7f63f002f240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:25.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.962+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:00:25.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.962+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:25.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0065420 con 0x7f6400146050 2021-03-09T09:00:25.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.963+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:25.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.963+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:25.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.963+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:25.979 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:25.980 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:25.980 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:25.980 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:25.981 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:25.981 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:25.982 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:25.982 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:25.983 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:25.983 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:25.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0065240 con 0x7f6400146050 2021-03-09T09:00:25.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.964+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:25.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.964+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:00:25.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:25.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f6400146050 2021-03-09T09:00:26.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:26.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:26.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:26.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:27.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:27.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:27.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:27.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:28.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:28.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:28.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:28.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:29.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:29.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:29.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:29.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:29.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:29.478+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:30.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:30.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:00:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bb20 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 rx=0x7f63e0025dc0 tx=0x7f63f002f240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bb20 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 rx=0x7f63e0025dc0 tx=0x7f63f002f240).stop 2021-03-09T09:00:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bb20 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:00:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065930 con 0x7f63e00070a0 2021-03-09T09:00:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:30.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078830 con 0x7f6400145bd0 2021-03-09T09:00:30.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:30.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:00:30.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:30.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:30.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:30.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.962+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:30.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.962+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:30.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:30.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.962+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:00:30.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.962+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:30.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.962+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:30.971 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:30.961+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5076 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:30.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.962+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:30.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.962+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.962+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:30.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:30.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:00:30.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:00:30.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:30.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:30.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bf50 con 0x7f63e0064730 2021-03-09T09:00:30.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:30.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f180 con 0x7f63e0064730 2021-03-09T09:00:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:00:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.963+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00016b0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7f63fc04d3a0 tx=0x7f63fc046ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.965+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:00:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.965+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T09:00:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.966+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.966+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:30.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.966+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:30.978 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:30.978 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:30.978 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:30.979 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:30.979 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:30.979 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:30.979 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:30.979 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:30.980 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:30.980 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:30.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0064730 2021-03-09T09:00:30.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.967+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:30.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T09:00:30.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:30.967+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:00:31.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:31.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:31.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:31.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:32.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:32.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:32.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:32.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:32.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:32.430+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:33.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:33.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:33.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:33.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:34.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:34.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:34.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:34.479+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:35.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:35.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:35.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00016b0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7f63fc04d3a0 tx=0x7f63fc046ae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00016b0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 rx=0x7f63fc04d3a0 tx=0x7f63fc046ae0).stop 2021-03-09T09:00:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:35.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00016b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:35.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00016b0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:35.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:00:35.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:35.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:35.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:35.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:35.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:35.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010730 con 0x7f63e00070a0 2021-03-09T09:00:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bb40 con 0x7f6400145bd0 2021-03-09T09:00:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000da50 con 0x7f6400146050 2021-03-09T09:00:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:35.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.962+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.963+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.963+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:00:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.963+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.963+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:35.973 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:35.963+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5119 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:35.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.963+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:35.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.963+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:35.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.963+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:35.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:35.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:35.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:35.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:35.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:35.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:00:35.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:00:35.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:35.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:35.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006cd30 con 0x7f6400146050 2021-03-09T09:00:35.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:35.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:35.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bf50 con 0x7f6400146050 2021-03-09T09:00:35.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:00:35.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.964+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:35.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.965+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc032f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:35.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.965+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:00:35.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.965+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:35.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00f140 con 0x7f6400146050 2021-03-09T09:00:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.966+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.966+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.966+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:35.983 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:35.983 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:35.983 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:35.983 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:35.984 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:35.984 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:35.984 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:35.984 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:35.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00f140 con 0x7f6400146050 2021-03-09T09:00:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.966+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.966+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:00:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:35.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049910 con 0x7f6400146050 2021-03-09T09:00:36.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:36.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:36.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:36.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:37.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:37.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:37.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:37.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:38.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:38.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:38.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:38.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:39.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:39.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:39.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:39.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:39.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:39.480+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:40.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:40.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:40.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:40.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:40.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc032f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:40.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc032f90).stop 2021-03-09T09:00:40.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:40.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:40.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0065bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:40.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:40.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:00:40.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:40.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:40.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00120a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:40.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.964+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:40.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:40.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:40.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:40.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f9f0 con 0x7f63e00070a0 2021-03-09T09:00:40.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:40.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006dd20 con 0x7f6400145bd0 2021-03-09T09:00:40.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:40.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63e0064730 2021-03-09T09:00:40.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:40.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:40.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:40.973 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5166 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:40.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:40.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.965+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:40.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.966+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00120a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:40.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.966+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:00:40.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.966+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:40.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.966+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:40.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.966+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:40.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.966+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:40.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.966+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:40.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:40.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:40.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:00:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:00:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00656e0 con 0x7f63e0064730 2021-03-09T09:00:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006cd30 con 0x7f63e0064730 2021-03-09T09:00:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:00:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.967+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.968+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:00:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.968+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00120a0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc067010).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.968+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0064730 2021-03-09T09:00:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.969+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.969+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.969+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:40.985 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:40.985 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:40.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0064730 2021-03-09T09:00:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.969+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.969+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:00:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:40.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0641a0 con 0x7f63e0064730 2021-03-09T09:00:41.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:41.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:41.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:41.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:42.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:42.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:42.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:42.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:42.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:42.431+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:43.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:43.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:43.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:43.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:44.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:44.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:44.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:44.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:44.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:44.481+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:45.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:45.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00120a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:00:45.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00120a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00120a0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc067010).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00120a0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc067010).stop 2021-03-09T09:00:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:45.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:45.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00120a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:45.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00120a0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:45.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:00:45.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:45.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:45.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:45.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:45.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:45.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065930 con 0x7f63e00070a0 2021-03-09T09:00:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:45.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e660 con 0x7f6400145bd0 2021-03-09T09:00:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:45.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027000 con 0x7f6400146050 2021-03-09T09:00:45.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:45.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:45.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:45.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:45.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:45.978 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:45.966+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5205 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:45.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:45.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:00:45.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:45.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:45.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:45.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:45.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:45.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:45.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:00:45.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:00:45.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:45.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:45.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f180 con 0x7f6400146050 2021-03-09T09:00:45.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:45.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00656e0 con 0x7f6400146050 2021-03-09T09:00:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:00:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.967+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.968+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7f63e00173d0 tx=0x7f63fc047590).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.968+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:00:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.968+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc077360 con 0x7f6400146050 2021-03-09T09:00:45.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.969+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:45.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.969+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:45.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.969+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:45.991 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:45.993 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f6400146050 2021-03-09T09:00:45.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.969+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:45.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.969+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:00:45.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:45.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065d50 con 0x7f6400146050 2021-03-09T09:00:46.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:46.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:46.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:46.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:47.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:47.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:47.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:47.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:48.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:48.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:48.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:48.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:49.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:49.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:49.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:49.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:49.482+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:50.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:50.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:50.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:00:50.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:50.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7f63e00173d0 tx=0x7f63fc047590).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 rx=0x7f63e00173d0 tx=0x7f63fc047590).stop 2021-03-09T09:00:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:50.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0065bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:50.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065bd0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:50.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:00:50.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:50.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:50.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:50.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:50.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:50.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:50.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:50.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e00070a0 2021-03-09T09:00:50.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:50.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010730 con 0x7f6400145bd0 2021-03-09T09:00:50.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:50.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e130 con 0x7f63e0064730 2021-03-09T09:00:50.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:50.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:50.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:50.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:50.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.967+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:50.976 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:50.968+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5232 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:50.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.969+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:50.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.969+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:00:50.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.969+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:50.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.969+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:50.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.969+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:50.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.969+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:50.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.969+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:50.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:50.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:50.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:50.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:50.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:50.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:00:50.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:00:50.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:50.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:50.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015370 con 0x7f63e0064730 2021-03-09T09:00:50.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:50.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:50.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006db20 con 0x7f63e0064730 2021-03-09T09:00:50.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:00:50.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.970+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:50.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.971+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:00:50.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.971+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:50.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bb20 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f63f0065810 tx=0x7f63f00652b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:50.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.972+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0064730 2021-03-09T09:00:50.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.972+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:50.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.972+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:50.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.972+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:50.984 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:50.984 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:50.984 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:50.984 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:50.984 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:50.985 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:50.985 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:50.985 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:50.985 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:50.985 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:50.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0064730 2021-03-09T09:00:50.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.976+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:50.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.976+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:00:50.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:50.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f63e0064730 2021-03-09T09:00:51.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:51.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:51.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:51.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:52.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:52.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:52.432+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:53.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:53.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:53.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:53.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:54.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:54.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:54.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:54.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:54.483+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:00:55.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:55.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:55.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:00:55.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:00:55.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bb20 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f63f0065810 tx=0x7f63f00652b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:00:55.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bb20 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 rx=0x7f63f0065810 tx=0x7f63f00652b0).stop 2021-03-09T09:00:55.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.968+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:55.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.968+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:00:55.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002bb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:00:55.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002bb20 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:00:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:00:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:00:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:55.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:00:55.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:00:55.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:00:55.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:55.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009470 con 0x7f63e00070a0 2021-03-09T09:00:55.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:55.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:00:55.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:00:55.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013280 con 0x7f6400146050 2021-03-09T09:00:55.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:00:55.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:00:55.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:55.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:00:55.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:00:55.977 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5279 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:00:55.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.969+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:00:55.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:00:55.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:00:55.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:00:55.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:00:55.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:00:55.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:00:55.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:00:55.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:55.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:55.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:00:55.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:00:55.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:00:55.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:00:55.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:00:55.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:55.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00106e0 con 0x7f6400146050 2021-03-09T09:00:55.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:00:55.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:00:55.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015370 con 0x7f6400146050 2021-03-09T09:00:55.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.971+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:00:55.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.971+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:55.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.972+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:00:55.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.972+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:00:55.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.972+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7f63e0016e10 tx=0x7f63f00540b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:00:55.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.974+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:00:55.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.975+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:00:55.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.975+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:00:55.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.975+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:00:55.985 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:00:55.986 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:00:55.986 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:55.986 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:00:55.986 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:00:55.986 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:00:55.987 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:00:55.987 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:00:55.988 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:00:55.988 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:00:55.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004e700 con 0x7f6400146050 2021-03-09T09:00:55.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.976+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:00:55.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.976+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:00:55.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:55.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f6400146050 2021-03-09T09:00:56.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:56.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:56.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:56.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:57.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:57.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:57.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:57.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:58.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:58.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:58.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:58.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:59.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:59.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:00:59.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:59.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:00:59.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:00:59.484+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:00.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:00.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.969+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:00.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:00.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7f63e0016e10 tx=0x7f63f00540b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:00.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 rx=0x7f63e0016e10 tx=0x7f63f00540b0).stop 2021-03-09T09:01:00.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:00.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:00.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:00.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:00.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:01:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:00.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:00.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:00.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:00.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:00.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63e00070a0 2021-03-09T09:01:00.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:00.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f6400145bd0 2021-03-09T09:01:00.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:00.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ecb0 con 0x7f63e0064730 2021-03-09T09:01:00.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:00.977 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5305 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:00.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:00.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:00.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:00.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:00.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:00.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:01:00.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:00.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.970+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:00.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:00.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:00.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:00.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:00.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:00.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:00.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:00.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:00.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:01:00.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:01:00.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:00.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:00.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0078370 con 0x7f63e0064730 2021-03-09T09:01:00.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:00.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:00.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f180 con 0x7f63e0064730 2021-03-09T09:01:00.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:01:00.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.971+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:00.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.972+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:01:00.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.972+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:00.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.972+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000baa0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 rx=0x7f63fc077520 tx=0x7f63fc047670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:00.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04f000 con 0x7f63e0064730 2021-03-09T09:01:00.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.973+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:00.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.973+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:00.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.973+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:00.986 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:00.986 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:00.987 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:00.987 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:00.987 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:00.987 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:00.988 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0064730 2021-03-09T09:01:00.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.973+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:00.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.973+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:01:00.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:00.974+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f63e0064730 2021-03-09T09:01:01.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:01.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:01.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:01.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:02.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:02.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:02.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:02.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:02.434+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:03.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:03.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:03.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:03.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:04.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:04.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:04.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:04.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:04.484+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:05.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:05.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.969+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:05.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.969+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.969+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000baa0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 rx=0x7f63fc077520 tx=0x7f63fc047670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.969+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000baa0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 rx=0x7f63fc077520 tx=0x7f63fc047670).stop 2021-03-09T09:01:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.969+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:05.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.969+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000baa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000baa0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:01:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:05.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:05.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:05.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:05.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:05.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:05.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:05.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b830 con 0x7f63e00070a0 2021-03-09T09:01:05.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:05.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e130 con 0x7f6400145bd0 2021-03-09T09:01:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:01:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:05.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:05.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:01:05.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:05.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:05.980 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:05.970+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5353 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:05.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:05.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:05.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:05.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:05.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:05.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:05.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:05.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:05.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:01:05.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:01:05.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:05.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:05.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024100 con 0x7f6400146050 2021-03-09T09:01:05.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:05.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:05.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0078370 con 0x7f6400146050 2021-03-09T09:01:05.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:01:05.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.971+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:05.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.972+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:01:05.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.972+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023c10 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc02c290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:05.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.972+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:05.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00f5a0 con 0x7f6400146050 2021-03-09T09:01:05.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.973+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.973+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.973+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:05.987 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:05.989 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00f5a0 con 0x7f6400146050 2021-03-09T09:01:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.973+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.973+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:01:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:05.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc051d10 con 0x7f6400146050 2021-03-09T09:01:06.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:06.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:06.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:06.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:07.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:07.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:07.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:07.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:08.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:08.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:08.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:08.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:09.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:09.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:09.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:09.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:09.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:09.485+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:10.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:10.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:10.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:01:10.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:10.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023c10 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc02c290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:10.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023c10 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc02c290).stop 2021-03-09T09:01:10.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:10.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:10.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:10.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023c10 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:10.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:01:10.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:10.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:10.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:10.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:10.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:10.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:10.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:10.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e660 con 0x7f63e00070a0 2021-03-09T09:01:10.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:10.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a980 con 0x7f6400145bd0 2021-03-09T09:01:10.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:10.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e0064730 2021-03-09T09:01:10.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:10.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:10.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:10.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:10.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:01:10.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:10.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:10.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:10.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:10.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:10.992 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:10.971+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5378 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:10.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:10.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:10.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:10.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:10.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:10.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:10.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:10.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:01:10.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:01:10.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:10.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:10.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001aa50 con 0x7f63e0064730 2021-03-09T09:01:10.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:10.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018890 con 0x7f63e0064730 2021-03-09T09:01:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:01:11.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.972+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:11.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.973+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:01:11.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.973+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:11.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.973+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001fb50 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f63f405b610 tx=0x7f63f402af00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:11.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f63e0064730 2021-03-09T09:01:11.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.975+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:11.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.975+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:11.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.975+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:11.004 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:11.005 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:11.005 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:11.008 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:11.008 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4018350 con 0x7f63e0064730 2021-03-09T09:01:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.975+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:11.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.975+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:01:11.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:10.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f63e0064730 2021-03-09T09:01:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:11.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:11.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:12.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:12.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:12.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:12.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:12.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:12.434+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:13.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:13.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:13.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:13.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:14.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:14.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:14.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:14.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:14.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:14.486+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:15.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:15.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:15.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:15.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:15.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001fb50 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f63f405b610 tx=0x7f63f402af00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:15.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001fb50 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 rx=0x7f63f405b610 tx=0x7f63f402af00).stop 2021-03-09T09:01:15.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:15.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:15.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001fb50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:15.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001fb50 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:15.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:01:15.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:15.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:15.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:15.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016f30 con 0x7f63e00070a0 2021-03-09T09:01:15.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:15.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024880 con 0x7f6400145bd0 2021-03-09T09:01:15.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:15.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ecb0 con 0x7f6400146050 2021-03-09T09:01:15.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:15.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:15.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:15.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:15.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.972+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:15.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:15.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:01:15.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:15.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:15.982 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5430 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:15.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:15.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:15.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:15.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:15.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:15.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:15.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:15.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:15.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:01:15.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:01:15.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:15.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:15.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f6400146050 2021-03-09T09:01:15.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:15.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:15.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024100 con 0x7f6400146050 2021-03-09T09:01:15.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:01:15.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.973+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:15.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.974+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cb20 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7f63fc00f300 tx=0x7f63fc02bf10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:15.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.974+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:01:15.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.974+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:15.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.974+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc076360 con 0x7f6400146050 2021-03-09T09:01:15.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.974+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:15.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.974+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:15.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.974+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:15.988 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:15.989 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:15.989 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:15.989 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:15.989 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:15.989 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:15.990 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:15.990 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:15.990 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:15.990 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:15.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc076130 con 0x7f6400146050 2021-03-09T09:01:15.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.975+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:15.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.975+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:01:15.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:15.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f6400146050 2021-03-09T09:01:16.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:16.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:16.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:16.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:17.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:17.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:17.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:17.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:18.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:18.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:18.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:18.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:19.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:19.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:19.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:19.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:19.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:19.487+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:20.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:20.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:20.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cb20 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7f63fc00f300 tx=0x7f63fc02bf10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cb20 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 rx=0x7f63fc00f300 tx=0x7f63fc02bf10).stop 2021-03-09T09:01:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:20.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007cb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007cb20 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:01:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:20.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:20.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e130 con 0x7f63e00070a0 2021-03-09T09:01:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a980 con 0x7f6400145bd0 2021-03-09T09:01:20.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001dcf0 con 0x7f63e0064730 2021-03-09T09:01:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:20.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:20.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:20.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:01:20.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:20.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.973+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:20.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:20.983 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5469 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:20.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:20.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:20.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:01:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:01:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:20.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T09:01:20.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:20.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:20.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00106e0 con 0x7f63e0064730 2021-03-09T09:01:20.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:01:20.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.974+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:20.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d010 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f63e00173d0 tx=0x7f63f0054360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:20.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:01:20.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:20.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006d90 con 0x7f63e0064730 2021-03-09T09:01:20.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:20.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:20.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:20.990 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:20.991 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:20.991 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:20.991 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:20.991 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:20.991 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:20.992 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:20.992 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:20.992 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:20.992 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003f3b0 con 0x7f63e0064730 2021-03-09T09:01:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.975+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:01:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:20.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T09:01:21.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:21.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:21.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:22.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:22.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:22.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:22.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:22.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:22.435+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:23.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:23.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:23.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:23.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:24.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:24.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:24.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:24.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:24.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:24.488+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:25.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:25.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:25.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:01:25.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:25.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d010 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f63e00173d0 tx=0x7f63f0054360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:25.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d010 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 rx=0x7f63e00173d0 tx=0x7f63f0054360).stop 2021-03-09T09:01:25.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:25.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:25.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007d010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:25.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007d010 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:25.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:01:25.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:25.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:25.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:25.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:25.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:25.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:25.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:25.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000da50 con 0x7f63e00070a0 2021-03-09T09:01:25.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:25.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002eb50 con 0x7f6400145bd0 2021-03-09T09:01:25.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:25.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024880 con 0x7f6400146050 2021-03-09T09:01:25.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:25.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:25.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:25.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:25.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.974+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:25.985 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:25.975+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5506 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:25.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.975+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:25.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.975+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:01:25.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.975+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:25.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.975+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:25.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:25.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:25.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:25.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:25.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:25.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:25.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:25.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:25.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:01:25.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:01:25.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:25.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:25.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00669d0 con 0x7f6400146050 2021-03-09T09:01:25.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:25.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:25.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001aa50 con 0x7f6400146050 2021-03-09T09:01:25.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:01:25.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.976+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:25.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.977+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:01:25.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.977+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f63f4034780 tx=0x7f63f40037c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:25.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.977+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:25.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f6400146050 2021-03-09T09:01:25.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.978+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:25.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.978+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:25.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.978+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:25.993 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:25.994 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:25.994 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:25.994 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:25.994 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:25.995 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:25.995 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:25.995 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:25.995 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:25.995 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:25.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4018350 con 0x7f6400146050 2021-03-09T09:01:25.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.978+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:25.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.978+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:01:25.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:25.979+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:01:26.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:26.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:26.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:26.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:27.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:27.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:27.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:27.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:28.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:28.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:28.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:28.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:29.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:29.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:29.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:29.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:29.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:29.489+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:30.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:30.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:30.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.975+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:30.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.975+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:30.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.975+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f63f4034780 tx=0x7f63f40037c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:30.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 rx=0x7f63f4034780 tx=0x7f63f40037c0).stop 2021-03-09T09:01:30.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:30.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:30.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021b60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:30.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:30.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:01:30.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:30.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:30.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:30.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:30.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:30.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:30.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:30.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00279b0 con 0x7f63e00070a0 2021-03-09T09:01:30.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:30.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:01:30.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:30.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e130 con 0x7f63e0064730 2021-03-09T09:01:30.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:30.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:30.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:30.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:30.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:30.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:30.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.977+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:01:30.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.977+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:30.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.977+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:30.987 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:30.976+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5552 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:30.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.977+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:30.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.977+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:30.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.977+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:30.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:30.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:30.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:30.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:30.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:30.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:01:30.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:01:30.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:30.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:30.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e430 con 0x7f63e0064730 2021-03-09T09:01:30.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:30.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:30.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00669d0 con 0x7f63e0064730 2021-03-09T09:01:30.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:01:30.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.978+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:30.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.979+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:01:30.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.979+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:30.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.979+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007cb20 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f63f40038e0 tx=0x7f63f4070170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:30.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40186d0 con 0x7f63e0064730 2021-03-09T09:01:30.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.980+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:30.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.980+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:30.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.980+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:30.999 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:30.999 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:31.000 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:31.000 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:31.000 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:31.000 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:31.000 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:31.001 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:31.001 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:31.001 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:31.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40186d0 con 0x7f63e0064730 2021-03-09T09:01:31.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.980+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:31.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.980+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:01:31.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:30.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f63e0064730 2021-03-09T09:01:31.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:31.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:31.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:31.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:32.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:32.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:32.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:32.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:32.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:32.436+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:33.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:33.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:33.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:33.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:34.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:34.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:34.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:34.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:34.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:34.490+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:35.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:35.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:35.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007cb20 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f63f40038e0 tx=0x7f63f4070170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:35.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007cb20 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 rx=0x7f63f40038e0 tx=0x7f63f4070170).stop 2021-03-09T09:01:35.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:35.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007cb20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007cb20 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:01:35.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:35.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:35.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:35.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:35.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:35.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:35.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:35.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c5a0 con 0x7f63e00070a0 2021-03-09T09:01:35.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:35.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00167f0 con 0x7f6400145bd0 2021-03-09T09:01:35.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:35.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c550 con 0x7f6400146050 2021-03-09T09:01:35.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:35.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:35.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:35.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:35.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.977+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:35.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.978+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:35.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.978+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:01:35.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.978+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:35.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.978+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:35.995 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:35.978+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5582 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:35.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.978+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:35.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.978+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:35.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.978+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:36.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:36.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:36.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:36.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:36.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:36.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:01:36.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:01:36.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:36.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:36.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c300 con 0x7f6400146050 2021-03-09T09:01:36.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:36.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:36.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:01:36.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:01:36.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.979+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:36.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.980+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f63f005cd00 tx=0x7f63f0054190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:36.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.980+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:01:36.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.980+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:36.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005d030 con 0x7f6400146050 2021-03-09T09:01:36.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.981+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:36.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.981+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:36.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.981+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:36.009 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:36.010 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:36.010 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:36.010 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:36.011 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:36.011 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:36.013 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005cb40 con 0x7f6400146050 2021-03-09T09:01:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.982+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.982+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:01:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:35.983+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T09:01:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:36.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:36.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:37.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:37.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:37.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:37.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:38.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:38.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:38.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:38.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:39.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:39.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:39.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:39.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:39.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:39.491+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:40.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:40.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f63f005cd00 tx=0x7f63f0054190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 rx=0x7f63f005cd00 tx=0x7f63f0054190).stop 2021-03-09T09:01:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021b60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021b60 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:01:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:40.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:40.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000dc50 con 0x7f63e00070a0 2021-03-09T09:01:40.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:40.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024880 con 0x7f6400145bd0 2021-03-09T09:01:40.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002eb50 con 0x7f63e0064730 2021-03-09T09:01:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.978+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:40.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:40.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:01:40.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:40.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:40.988 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5624 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:40.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:40.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:40.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:40.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:40.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.979+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:40.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:40.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:40.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:40.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:40.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:40.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:01:40.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:01:40.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:40.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:40.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001aa50 con 0x7f63e0064730 2021-03-09T09:01:40.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:40.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:40.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63e0064730 2021-03-09T09:01:40.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:01:40.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.980+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:40.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.981+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007ca50 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc006d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:40.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.981+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:01:40.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.981+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:40.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T09:01:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.982+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.982+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.982+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:40.996 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:40.996 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:40.996 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:40.996 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:40.997 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:40.997 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:40.997 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:40.997 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:40.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.983+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T09:01:40.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.983+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:40.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.983+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:01:40.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:40.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0064730 2021-03-09T09:01:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:41.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:41.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:42.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:42.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:42.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:42.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:42.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:42.438+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:43.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:43.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:43.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:43.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:44.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:44.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:44.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:44.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:44.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:44.492+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.979+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:01:45.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.979+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:45.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.979+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007ca50 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc006d40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:45.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.979+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007ca50 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc006d40).stop 2021-03-09T09:01:45.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.979+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:45.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:45.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007ca50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:45.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007ca50 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:45.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:01:45.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:45.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:45.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:45.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010130 con 0x7f63e00070a0 2021-03-09T09:01:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:45.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:01:45.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015380 con 0x7f6400146050 2021-03-09T09:01:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:45.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:45.995 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:45.980+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5657 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.981+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.982+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:01:46.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.982+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:46.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.982+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:01:46.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:01:46.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:46.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:46.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f6400146050 2021-03-09T09:01:46.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:46.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:46.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c300 con 0x7f6400146050 2021-03-09T09:01:46.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:01:46.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.983+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:46.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.984+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:01:46.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.984+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:46.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.984+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7f63f000ea70 tx=0x7f63f0013b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:46.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006d90 con 0x7f6400146050 2021-03-09T09:01:46.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.986+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.986+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.986+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:46.012 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:46.012 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:46.013 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:46.013 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:46.013 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:46.015 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:46.015 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:46.015 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:46.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006d90 con 0x7f6400146050 2021-03-09T09:01:46.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.987+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:46.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.987+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:01:46.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:45.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T09:01:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:46.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:46.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:46.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:47.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:47.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:47.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:47.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:48.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:48.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:48.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:48.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:49.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:49.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:49.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:49.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:49.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:49.493+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:50.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:50.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:50.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00202b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:01:50.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00202b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:50.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7f63f000ea70 tx=0x7f63f0013b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:50.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 rx=0x7f63f000ea70 tx=0x7f63f0013b10).stop 2021-03-09T09:01:50.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:50.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:50.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00202b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:50.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:50.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:01:50.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:50.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:50.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:50.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:50.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:50.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:50.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:50.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c1c0 con 0x7f63e00070a0 2021-03-09T09:01:50.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:50.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e760 con 0x7f6400145bd0 2021-03-09T09:01:50.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:50.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002eb50 con 0x7f63e0064730 2021-03-09T09:01:50.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:50.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:50.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:50.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.981+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:50.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.982+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:50.990 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:50.982+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5700 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:50.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.982+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:50.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.982+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:01:50.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.982+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:50.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.982+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:50.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.982+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:50.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.982+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:50.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.982+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:50.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:50.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:50.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:50.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:50.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:50.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:01:50.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:01:50.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:50.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:50.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001cd50 con 0x7f63e0064730 2021-03-09T09:01:50.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:50.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:50.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001aa50 con 0x7f63e0064730 2021-03-09T09:01:50.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:01:50.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.983+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:50.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.984+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:01:50.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:50.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.984+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:50.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.985+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T09:01:50.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.986+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:50.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.986+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:50.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.986+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:50.999 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:50.999 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:50.999 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:50.999 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:50.999 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:51.000 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:51.000 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:51.000 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:51.000 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:51.001 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:51.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T09:01:51.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.987+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:51.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.987+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:01:51.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:50.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0064730 2021-03-09T09:01:51.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:51.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:51.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:51.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:52.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:52.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:52.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:52.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:52.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:52.439+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:53.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:53.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:53.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:53.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:54.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:54.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:54.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:54.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:54.494+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:01:55.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:55.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:55.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:01:55.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:01:55.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:01:55.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a740).stop 2021-03-09T09:01:55.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:55.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:01:55.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:01:55.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:01:55.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:01:55.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:55.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:01:55.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:55.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:01:55.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:01:55.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:01:55.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:55.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cfa0 con 0x7f63e00070a0 2021-03-09T09:01:55.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:55.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e860 con 0x7f6400145bd0 2021-03-09T09:01:55.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:01:55.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e130 con 0x7f6400146050 2021-03-09T09:01:55.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.982+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:01:55.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.983+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:01:55.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.983+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:55.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:01:55.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.983+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:01:55.992 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:01:55.983+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5736 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:01:55.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:01:55.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.984+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:01:55.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.984+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:01:55.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.984+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:01:55.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.984+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:01:55.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.984+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:01:55.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.984+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:01:55.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:01:55.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:01:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:01:55.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:01:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:01:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:01:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:01:55.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:01:55.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:01:55.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001cd50 con 0x7f6400146050 2021-03-09T09:01:55.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:01:55.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.985+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:55.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.986+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f63fc04d760 tx=0x7f63fc046b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:01:55.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.986+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:01:55.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.986+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:01:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0470b0 con 0x7f6400146050 2021-03-09T09:01:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.987+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:01:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.987+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:01:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.987+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:01:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:01:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:01:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:01:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:01:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:01:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:01:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:01:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:01:56.003 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:01:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d530 con 0x7f6400146050 2021-03-09T09:01:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.989+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:01:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.989+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:01:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:55.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f6400146050 2021-03-09T09:01:56.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:56.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:56.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:56.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:57.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:57.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:57.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:57.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:58.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:58.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:58.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:58.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:59.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:59.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:01:59.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:59.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:01:59.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:01:59.495+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:00.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:00.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:00.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00202b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:00.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00202b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:00.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f63fc04d760 tx=0x7f63fc046b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:00.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 rx=0x7f63fc04d760 tx=0x7f63fc046b50).stop 2021-03-09T09:02:00.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00202b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:00.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:02:00.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:00.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:00.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:00.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:00.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:00.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:00.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:00.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:02:00.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:00.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012980 con 0x7f6400145bd0 2021-03-09T09:02:00.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:00.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e130 con 0x7f63e0064730 2021-03-09T09:02:00.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:00.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:00.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:00.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:00.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.984+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:00.993 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:00.985+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5773 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:00.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.985+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:00.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.985+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:02:00.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.985+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:00.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.985+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:00.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.985+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:00.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.985+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:00.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.985+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:00.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:00.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:00.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:00.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:00.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:00.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:02:00.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:02:00.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:00.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:00.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f7c0 con 0x7f63e0064730 2021-03-09T09:02:00.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:00.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:00.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63e0064730 2021-03-09T09:02:00.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:02:00.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:00.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:02:00.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f000ef00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:00.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.986+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0064730 2021-03-09T09:02:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.988+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.988+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.988+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:01.000 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:01.001 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:01.001 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:01.001 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:01.001 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:01.001 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:01.002 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:01.002 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:01.002 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:01.003 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0064730 2021-03-09T09:02:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.988+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.988+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:02:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:00.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T09:02:01.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:01.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:01.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:01.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:02.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:02.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:02.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:02.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:02.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:02.441+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:03.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:03.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:03.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:03.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:04.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:04.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:04.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:04.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:04.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:04.496+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:05.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:05.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:05.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f000ef00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f000ef00).stop 2021-03-09T09:02:05.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:05.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:02:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:05.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00777a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002eb50 con 0x7f63e00070a0 2021-03-09T09:02:05.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:05.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e760 con 0x7f6400145bd0 2021-03-09T09:02:05.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:05.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00223b0 con 0x7f6400146050 2021-03-09T09:02:05.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:05.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:05.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:05.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:05.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.985+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:05.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00777a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:05.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:02:05.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:05.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:05.996 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5812 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:05.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:05.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:05.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:05.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:06.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:06.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:06.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:06.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:06.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:02:06.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:02:06.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:06.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:06.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006db70 con 0x7f6400146050 2021-03-09T09:02:06.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:06.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:06.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f7c0 con 0x7f6400146050 2021-03-09T09:02:06.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:02:06.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.986+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:06.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.987+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:02:06.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00777a0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f63f001b010 tx=0x7f63f0006910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:06.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.987+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001abb0 con 0x7f6400146050 2021-03-09T09:02:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.988+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.988+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.988+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:06.006 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:06.006 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:06.006 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:06.006 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:06.006 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:06.007 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001abb0 con 0x7f6400146050 2021-03-09T09:02:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.989+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.989+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:02:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:05.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f6400146050 2021-03-09T09:02:06.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:06.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:06.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:06.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:07.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:07.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:07.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:07.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:08.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:08.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:08.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:08.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:09.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:09.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:09.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:09.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:09.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:09.497+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:10.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:10.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:10.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00777a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:10.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00777a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:10.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00777a0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f63f001b010 tx=0x7f63f0006910).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:10.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00777a0 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 rx=0x7f63f001b010 tx=0x7f63f0006910).stop 2021-03-09T09:02:10.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:10.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:10.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00777a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:10.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00777a0 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:02:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:10.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:10.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:10.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:10.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:10.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:02:10.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:10.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002eb50 con 0x7f6400145bd0 2021-03-09T09:02:10.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:10.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001aa90 con 0x7f63e0064730 2021-03-09T09:02:10.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:10.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:10.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:10.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.986+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:10.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:10.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:10.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:02:10.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:10.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:10.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:10.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:10.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:10.998 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5849 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:10.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:10.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:10.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:10.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:10.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:02:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:02:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:11.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ff80 con 0x7f63e0064730 2021-03-09T09:02:11.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:11.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:11.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006db70 con 0x7f63e0064730 2021-03-09T09:02:11.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.988+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:02:11.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.988+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:11.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.988+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:02:11.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.988+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:11.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.988+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f63e0003380 tx=0x7f63f0030f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:11.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00537b0 con 0x7f63e0064730 2021-03-09T09:02:11.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.990+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:11.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.990+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:11.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.990+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:11.004 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:11.004 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:11.004 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:11.004 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:11.004 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:11.005 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:11.005 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:11.005 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:11.005 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:11.006 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001ade0 con 0x7f63e0064730 2021-03-09T09:02:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.990+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.990+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:02:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:10.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T09:02:11.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:11.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:11.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:11.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:12.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:12.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:12.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:12.442+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:13.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:13.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:13.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:13.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:14.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:14.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:14.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:14.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:14.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:14.498+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:15.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:15.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:15.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:15.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:15.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f63e0003380 tx=0x7f63f0030f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:15.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 rx=0x7f63e0003380 tx=0x7f63f0030f30).stop 2021-03-09T09:02:15.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:15.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:15.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:15.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:15.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:02:15.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:15.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:15.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:15.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:15.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:15.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:15.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:15.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002eb50 con 0x7f63e00070a0 2021-03-09T09:02:15.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:15.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f950 con 0x7f6400145bd0 2021-03-09T09:02:15.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:15.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:02:15.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:15.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:15.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:15.996 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5874 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:15.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:15.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:15.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:15.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:02:15.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:15.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:15.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:15.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:16.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.988+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:16.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:16.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:16.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:16.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:16.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:16.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:02:16.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:02:16.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:16.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:16.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:02:16.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:16.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:16.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:02:16.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:02:16.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.989+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:16.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:02:16.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d9a0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7f63fc032750 tx=0x7f63fc047d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:16.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:16.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023770 con 0x7f6400146050 2021-03-09T09:02:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:16.005 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:16.006 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:16.006 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:16.006 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:16.006 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:16.006 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:16.007 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:16.007 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:16.007 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:16.007 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:16.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046f60 con 0x7f6400146050 2021-03-09T09:02:16.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:16.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.990+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:02:16.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:15.992+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T09:02:16.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:16.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:16.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:16.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:17.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:17.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:17.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:17.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:18.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:18.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:18.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:19.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:19.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:19.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:19.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:19.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:19.499+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:20.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:20.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:20.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.988+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:20.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.988+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:20.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.988+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d9a0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7f63fc032750 tx=0x7f63fc047d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:20.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.988+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d9a0 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 rx=0x7f63fc032750 tx=0x7f63fc047d30).stop 2021-03-09T09:02:20.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006d9a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d9a0 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:20.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:02:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:20.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:20.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002eb50 con 0x7f63e00070a0 2021-03-09T09:02:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078780 con 0x7f6400145bd0 2021-03-09T09:02:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f63e0064730 2021-03-09T09:02:20.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:20.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:02:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:20.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:20.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.989+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:20.999 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5910 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:21.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:21.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:21.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:21.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:21.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:02:21.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:02:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a070 con 0x7f63e0064730 2021-03-09T09:02:21.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:21.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:21.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00669d0 con 0x7f63e0064730 2021-03-09T09:02:21.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:02:21.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.990+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:21.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.991+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:02:21.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.991+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:21.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.991+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f63f4071da0 tx=0x7f63f40038a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:21.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.992+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4023d70 con 0x7f63e0064730 2021-03-09T09:02:21.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.992+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:21.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.992+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:21.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.992+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:21.007 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:21.007 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:21.007 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:21.009 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f63e0064730 2021-03-09T09:02:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.993+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.993+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:02:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:20.994+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T09:02:21.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:21.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:21.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:21.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:22.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:22.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:22.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:22.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:22.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:22.443+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:23.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:23.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:23.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:23.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:24.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:24.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:24.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:24.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:24.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:24.500+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:25.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:25.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:25.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:25.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:25.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f63f4071da0 tx=0x7f63f40038a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:25.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 rx=0x7f63f4071da0 tx=0x7f63f40038a0).stop 2021-03-09T09:02:25.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:25.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:25.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:25.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=CLOSED pgs=2263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:25.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:02:25.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:25.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:25.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:25.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:25.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:25.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:25.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:25.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015340 con 0x7f63e00070a0 2021-03-09T09:02:25.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:26.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:02:26.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:26.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e60 con 0x7f6400146050 2021-03-09T09:02:26.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:26.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:26.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:26.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:26.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:02:26.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:26.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:26.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.990+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:26.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:26.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:26.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:26.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:26.005 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5952 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:26.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:26.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:26.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:26.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:26.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:26.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:02:26.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:02:26.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:26.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:26.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:02:26.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:26.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:26.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ff80 con 0x7f6400146050 2021-03-09T09:02:26.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:02:26.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.991+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:26.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.992+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:02:26.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f005dfc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:26.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.992+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:26.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f6400146050 2021-03-09T09:02:26.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.993+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:26.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.993+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:26.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.993+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:26.014 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:26.014 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:26.014 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:26.015 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:26.015 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:26.015 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:26.016 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:26.016 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:26.017 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:26.017 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:26.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0000d20 con 0x7f6400146050 2021-03-09T09:02:26.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.993+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:26.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.993+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:02:26.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:25.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a050 con 0x7f6400146050 2021-03-09T09:02:26.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:26.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:26.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:26.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:27.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:27.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:27.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:27.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:28.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:28.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:28.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:28.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:29.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:29.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:29.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:29.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:29.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:29.501+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:30.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:30.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:30.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00202b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:30.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00202b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:30.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f005dfc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:30.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 secure :-1 s=READY pgs=2277 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f005dfc0).stop 2021-03-09T09:02:30.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:30.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:30.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00202b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:30.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00202b0 unknown :-1 s=CLOSED pgs=2277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:30.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:02:30.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:30.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:30.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:30.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:30.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:30.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:30.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:30.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025590 con 0x7f63e00070a0 2021-03-09T09:02:30.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:31.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:02:31.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:31.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f63e0064730 2021-03-09T09:02:31.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:31.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:31.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:31.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:31.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:31.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:31.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:02:31.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:31.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:31.006 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 5993 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:31.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:31.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:31.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:31.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:31.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:31.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:31.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:31.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:31.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:02:31.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:02:31.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:31.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:31.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028720 con 0x7f63e0064730 2021-03-09T09:02:31.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:31.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:31.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T09:02:31.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:02:31.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:31.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:02:31.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:31.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:31.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0048400 con 0x7f63e0064730 2021-03-09T09:02:31.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:31.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:31.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:31.020 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:31.021 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:31.021 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:31.021 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:31.022 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:31.022 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:31.023 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:31.023 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:31.023 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:31.023 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:31.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f006ca90 con 0x7f63e0064730 2021-03-09T09:02:31.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:31.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.993+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:02:31.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:30.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0053170 con 0x7f63e0064730 2021-03-09T09:02:31.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:31.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:31.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:31.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:32.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:32.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:32.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:32.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:32.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:32.444+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:33.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:33.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:33.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:33.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:34.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:34.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:34.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:34.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:34.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:34.502+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:35.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:35.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:35.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:35.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:35.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:35.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0000bc0).stop 2021-03-09T09:02:35.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:35.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:35.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:36.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:36.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:02:36.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:36.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:36.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:36.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:36.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:36.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:36.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:36.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ca90 con 0x7f63e00070a0 2021-03-09T09:02:36.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:36.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:02:36.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:36.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f6400146050 2021-03-09T09:02:36.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:36.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:36.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:36.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:36.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:36.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:36.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:02:36.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:36.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:36.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:36.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:36.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:36.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.993+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:36.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:36.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:36.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:36.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:36.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:02:36.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:02:36.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:36.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:36.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ff80 con 0x7f6400146050 2021-03-09T09:02:36.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:36.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:36.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a070 con 0x7f6400146050 2021-03-09T09:02:36.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:02:36.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:36.010 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6035 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:36.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f020 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4052290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:36.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:02:36.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.994+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:02:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.997+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.997+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.997+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:36.015 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40092a0 con 0x7f6400146050 2021-03-09T09:02:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.997+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.997+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:02:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:35.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:36.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:36.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:37.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:37.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:37.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:37.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:38.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:38.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:38.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:38.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:39.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:39.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:39.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:39.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:39.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:39.503+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:40.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:40.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:40.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.994+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:40.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.994+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:40.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.994+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f020 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4052290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:40.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.994+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f020 secure :-1 s=READY pgs=2303 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4052290).stop 2021-03-09T09:02:40.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001f020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f020 unknown :-1 s=CLOSED pgs=2303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:41.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:02:41.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:41.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:41.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:41.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:41.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0079d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:41.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:41.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:41.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cd20 con 0x7f63e00070a0 2021-03-09T09:02:41.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:41.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077640 con 0x7f6400145bd0 2021-03-09T09:02:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:02:41.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0079d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:41.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:02:41.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:41.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:41.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:41.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:41.007 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:40.995+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6074 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:41.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:41.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:41.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:41.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:41.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:41.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:41.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:41.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:41.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:02:41.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:02:41.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:41.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:41.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002eef0 con 0x7f63e0064730 2021-03-09T09:02:41.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:41.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:41.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028720 con 0x7f63e0064730 2021-03-09T09:02:41.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:02:41.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.996+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:41.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0079d40 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00484e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:41.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.997+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:02:41.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.997+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:41.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001a9e0 con 0x7f63e0064730 2021-03-09T09:02:41.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.999+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:41.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.999+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:41.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.999+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:41.017 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:41.017 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:41.017 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:41.018 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:41.018 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:41.018 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:41.018 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:41.019 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:41.019 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:41.019 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:41.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001a9e0 con 0x7f63e0064730 2021-03-09T09:02:41.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.999+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:41.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:40.999+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:02:41.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:41.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f63e0064730 2021-03-09T09:02:41.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:41.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:41.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:41.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:42.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:42.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:42.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:42.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:42.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:42.446+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:43.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:43.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:44.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:43.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:44.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:44.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:44.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:44.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:44.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:44.504+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:45.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:45.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0079d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:02:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0079d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0079d40 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00484e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:46.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0079d40 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00484e0).stop 2021-03-09T09:02:46.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:46.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:46.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0079d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:46.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0079d40 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:02:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077500 con 0x7f63e00070a0 2021-03-09T09:02:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00277b0 con 0x7f6400145bd0 2021-03-09T09:02:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:02:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.996+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.997+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:46.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.997+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:02:46.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.997+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:46.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.997+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:46.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.997+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:46.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.997+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:46.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.997+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:46.007 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:45.997+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6109 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:46.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:46.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:46.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:46.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:46.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:46.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:02:46.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:02:46.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:46.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:46.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c240 con 0x7f6400146050 2021-03-09T09:02:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ff80 con 0x7f6400146050 2021-03-09T09:02:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:02:46.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:46.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:02:46.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025200 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f405b5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:46.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.998+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:46.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f6400146050 2021-03-09T09:02:46.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.999+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:46.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.999+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:46.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.999+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:46.013 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:46.013 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:46.013 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:46.014 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:46.015 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:46.015 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:46.015 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:46.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b3a0 con 0x7f6400146050 2021-03-09T09:02:46.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.999+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:46.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:45.999+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:02:46.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:46.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:02:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:46.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:46.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:47.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:47.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:47.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:47.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:48.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:48.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:48.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:48.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:49.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:49.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:49.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:49.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:49.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:49.505+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:50.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:50.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:51.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:51.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:51.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025200 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f405b5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:51.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025200 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f405b5a0).stop 2021-03-09T09:02:51.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:51.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:51.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:51.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025200 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:51.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:02:51.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:51.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:51.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:51.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:51.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:51.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:51.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:51.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f63e00070a0 2021-03-09T09:02:51.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:51.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cd20 con 0x7f6400145bd0 2021-03-09T09:02:51.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:51.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00237d0 con 0x7f63e0064730 2021-03-09T09:02:51.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:51.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:51.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:51.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:51.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:51.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:51.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:02:51.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:51.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:51.011 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:50.998+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6140 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:51.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.999+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:51.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.999+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:51.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:50.999+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:51.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:51.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:51.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:51.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:51.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:51.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:02:51.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:02:51.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:51.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:51.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ff80 con 0x7f63e0064730 2021-03-09T09:02:51.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:51.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:51.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c240 con 0x7f63e0064730 2021-03-09T09:02:51.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:02:51.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:51.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007c3d0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f400d480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:51.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:02:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.000+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009350 con 0x7f63e0064730 2021-03-09T09:02:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.001+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.001+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.001+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:51.021 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:51.021 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:51.021 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:51.021 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:51.021 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:51.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e0064730 2021-03-09T09:02:51.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.004+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:51.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405b850 con 0x7f63e0064730 2021-03-09T09:02:51.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.004+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:02:51.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:51.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:51.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:52.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:52.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:52.447+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:53.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:53.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:53.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:53.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:54.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:54.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:54.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:54.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:54.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:54.506+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:02:55.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:55.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:02:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:02:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007c3d0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f400d480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:02:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007c3d0 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f400d480).stop 2021-03-09T09:02:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:02:56.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007c3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:02:56.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007c3d0 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:02:56.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:02:56.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:56.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:02:56.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:56.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:02:56.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:02:56.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:02:56.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:56.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:02:56.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:56.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f6400145bd0 2021-03-09T09:02:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:02:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f6400146050 2021-03-09T09:02:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:02:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:02:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:56.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:02:56.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:02:56.010 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:02:55.999+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6183 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:02:56.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.000+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:02:56.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.000+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:02:56.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.000+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:02:56.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.000+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:02:56.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.000+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:02:56.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.000+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:02:56.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.000+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:02:56.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:02:56.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:56.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:56.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:02:56.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:02:56.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:02:56.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:02:56.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:02:56.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:02:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:02:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:02:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ff80 con 0x7f6400146050 2021-03-09T09:02:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:02:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:02:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f63f4029f40 tx=0x7f63f4009370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:02:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.001+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:02:56.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4029d80 con 0x7f6400146050 2021-03-09T09:02:56.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.002+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:02:56.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.002+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:02:56.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.002+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:02:56.018 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:02:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:02:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:02:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:02:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:02:56.020 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:02:56.020 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:02:56.020 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:02:56.020 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:02:56.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4029b50 con 0x7f6400146050 2021-03-09T09:02:56.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.002+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:02:56.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.002+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:02:56.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f6400146050 2021-03-09T09:02:56.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:56.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:56.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:57.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:57.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:57.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:57.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:58.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:58.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:58.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:58.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:59.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:59.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:02:59.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:59.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:02:59.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:02:59.507+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:00.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:00.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:00.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:00.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:03:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:01.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f63f4029f40 tx=0x7f63f4009370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:01.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 rx=0x7f63f4029f40 tx=0x7f63f4009370).stop 2021-03-09T09:03:01.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:01.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:01.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:01.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:01.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:03:01.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:01.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:01.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:01.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:01.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:01.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:01.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:03:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cd20 con 0x7f6400145bd0 2021-03-09T09:03:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f63e0064730 2021-03-09T09:03:01.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:01.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:01.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:01.009 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6218 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:01.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.000+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.001+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:03:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.001+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.001+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.001+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.001+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.001+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:03:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:03:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00656f0 con 0x7f63e0064730 2021-03-09T09:03:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f820 con 0x7f63e0064730 2021-03-09T09:03:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:03:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.002+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.003+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024790 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 rx=0x7f63f400d740 tx=0x7f63f4063250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.003+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:03:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.003+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.005+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4023280 con 0x7f63e0064730 2021-03-09T09:03:01.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.005+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:01.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.005+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:01.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.005+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:01.017 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:01.017 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:01.019 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4023280 con 0x7f63e0064730 2021-03-09T09:03:01.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.008+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:01.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.008+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:03:01.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63e0064730 2021-03-09T09:03:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:01.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:02.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:02.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:02.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:02.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:02.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:02.449+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:03.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:03.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:03.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:03.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:04.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:04.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:04.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:04.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:04.508+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:05.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:05.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:05.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:05.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:06.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0024790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:06.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0024790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:06.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024790 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 rx=0x7f63f400d740 tx=0x7f63f4063250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:06.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024790 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 rx=0x7f63f400d740 tx=0x7f63f4063250).stop 2021-03-09T09:03:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0024790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0024790 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:06.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:03:06.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:06.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:06.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:06.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:06.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012b80 con 0x7f63e00070a0 2021-03-09T09:03:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:03:06.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f6400146050 2021-03-09T09:03:06.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:06.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:03:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:06.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.001+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:06.013 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6257 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:06.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:06.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:03:06.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:03:06.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:06.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:06.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:03:06.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:03:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:03:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.002+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.003+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:03:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.003+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.003+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a350 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc075f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c430 con 0x7f6400146050 2021-03-09T09:03:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.004+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.004+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.004+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:06.025 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02c430 con 0x7f6400146050 2021-03-09T09:03:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.004+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.004+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:03:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.005+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01fba0 con 0x7f6400146050 2021-03-09T09:03:06.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:06.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:06.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:07.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:07.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:07.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:07.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:08.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:08.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:08.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:08.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:09.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:09.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:09.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:09.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:09.509+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:10.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:10.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:10.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:10.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:11.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:11.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a350 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc075f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a350 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc075f70).stop 2021-03-09T09:03:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:11.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002a350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:11.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a350 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:11.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:03:11.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:11.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:11.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:11.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63e00070a0 2021-03-09T09:03:11.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:11.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:03:11.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:11.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022250 con 0x7f63e0064730 2021-03-09T09:03:11.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:11.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:11.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:11.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:11.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:11.012 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:11.003+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6293 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:11.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.004+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.004+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:03:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.004+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.004+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:11.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.005+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:11.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.005+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:11.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.005+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:11.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:11.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:03:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:03:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:11.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f820 con 0x7f63e0064730 2021-03-09T09:03:11.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:11.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:11.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T09:03:11.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:03:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:03:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.006+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc04fda0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:11.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc064600 con 0x7f63e0064730 2021-03-09T09:03:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.007+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.007+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.007+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:11.022 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc064600 con 0x7f63e0064730 2021-03-09T09:03:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.008+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.008+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:03:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049910 con 0x7f63e0064730 2021-03-09T09:03:11.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:11.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:11.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:12.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:12.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:12.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:12.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:12.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:12.450+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:13.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:13.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:13.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:13.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:14.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:14.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:14.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:14.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:14.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:14.510+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:15.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:15.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:15.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:15.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:16.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:03:16.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:16.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc04fda0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:16.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc04fda0).stop 2021-03-09T09:03:16.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:16.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:16.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e007bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:16.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e007bd50 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:16.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:03:16.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:16.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:16.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:16.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:16.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:16.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:16.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:16.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f700 con 0x7f63e00070a0 2021-03-09T09:03:16.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:16.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b410 con 0x7f6400145bd0 2021-03-09T09:03:16.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:16.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:03:16.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:16.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:16.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:16.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:16.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.004+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:16.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.005+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:16.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.005+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:03:16.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.005+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:16.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.005+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:16.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.005+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:16.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.005+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:16.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.005+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:16.016 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:16.005+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6329 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:16.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:16.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:16.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:16.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:16.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:16.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:03:16.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:03:16.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:16.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:16.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e910 con 0x7f6400146050 2021-03-09T09:03:16.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:16.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:16.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002eef0 con 0x7f6400146050 2021-03-09T09:03:16.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:03:16.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.006+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:16.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.007+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:03:16.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.007+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:16.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.007+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c240 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 rx=0x7f63f001a970 tx=0x7f63f004dd30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:16.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f6400146050 2021-03-09T09:03:16.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.008+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:16.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.008+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:16.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.008+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:16.022 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:16.022 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:16.023 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:16.023 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:16.023 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:16.023 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:16.023 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:16.024 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:16.024 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:16.024 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:16.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T09:03:16.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.008+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:16.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.008+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:03:16.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T09:03:16.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:16.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:16.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:17.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:17.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:17.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:17.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:18.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:18.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:18.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:18.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:19.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:19.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:19.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:19.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:19.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:19.511+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:20.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:20.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:20.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:20.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:03:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:21.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c240 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 rx=0x7f63f001a970 tx=0x7f63f004dd30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c240 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 rx=0x7f63f001a970 tx=0x7f63f004dd30).stop 2021-03-09T09:03:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:21.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c240 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:03:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:21.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:21.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022250 con 0x7f63e00070a0 2021-03-09T09:03:21.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:21.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b410 con 0x7f6400145bd0 2021-03-09T09:03:21.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:21.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:03:21.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:21.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:21.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:21.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:21.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:21.014 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6361 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:21.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:21.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:03:21.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:21.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:21.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:21.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:21.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.006+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:21.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:21.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:03:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:03:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0064730 2021-03-09T09:03:21.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e910 con 0x7f63e0064730 2021-03-09T09:03:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:03:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.007+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:21.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.008+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:03:21.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.008+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:21.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e680 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 rx=0x7f63f00090c0 tx=0x7f63f00554e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:21.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006d90 con 0x7f63e0064730 2021-03-09T09:03:21.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.009+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.009+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.009+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:21.022 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:21.023 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:21.023 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:21.023 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:21.023 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:21.023 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:21.024 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:21.024 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:21.024 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:21.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.011+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006d90 con 0x7f63e0064730 2021-03-09T09:03:21.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.011+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:21.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.011+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a040 con 0x7f63e0064730 2021-03-09T09:03:21.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.011+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:03:21.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:21.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:21.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:22.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:22.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:22.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:22.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:22.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:22.451+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:23.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:23.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:23.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:23.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:24.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:24.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:24.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:24.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:24.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:24.512+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:25.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:25.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:25.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:25.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:26.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:03:26.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:26.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e680 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 rx=0x7f63f00090c0 tx=0x7f63f00554e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:26.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e680 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 rx=0x7f63f00090c0 tx=0x7f63f00554e0).stop 2021-03-09T09:03:26.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:26.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:26.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000e680 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:26.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000e680 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:26.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:03:26.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:26.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:26.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:26.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:26.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:26.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:26.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:26.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b410 con 0x7f63e00070a0 2021-03-09T09:03:26.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:26.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078980 con 0x7f6400145bd0 2021-03-09T09:03:26.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:26.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f700 con 0x7f6400146050 2021-03-09T09:03:26.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:26.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:26.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.007+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:26.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:26.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:26.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:26.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:03:26.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:26.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:26.017 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6408 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:26.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:26.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:26.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.008+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:26.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:26.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:26.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:26.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:26.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:26.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:03:26.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:03:26.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:26.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:26.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001dd80 con 0x7f6400146050 2021-03-09T09:03:26.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:26.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:26.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:03:26.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:03:26.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:26.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c570 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7f63fc0099d0 tx=0x7f63fc049b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:26.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:03:26.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.009+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:26.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0641a0 con 0x7f6400146050 2021-03-09T09:03:26.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.010+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:26.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.010+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:26.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.010+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:26.023 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:26.023 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:26.023 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:26.024 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:26.024 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:26.024 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:26.024 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:26.024 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:26.025 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:26.025 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:26.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:03:26.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.010+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:26.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.011+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:03:26.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f6400146050 2021-03-09T09:03:26.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:26.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:26.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:27.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:27.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:27.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:27.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:28.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:28.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:28.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:28.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:29.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.341+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e005c7c0 msgr2=0x7f63e005ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000820) for more than 900000000 us, fault. 2021-03-09T09:03:29.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.341+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e005c7c0 0x7f63e005ec80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f63f00036d0 tx=0x7f63f0006b30).stop 2021-03-09T09:03:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e005c7c0 msgr2=0x7f63e005ec80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.341+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e005c7c0 0x7f63e005ec80 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.341+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 0x7f63e0017850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.341+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 0x7f63e0017850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.341+0000 7f6405d52700 10 monclient: get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:03:29.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.342+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 0x7f63e0017850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0006b30).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:29.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.342+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:03:29.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:29.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:29.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:29.513+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:30.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:30.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:30.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:30.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:31.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.008+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:31.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.008+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:31.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.008+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c570 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7f63fc0099d0 tx=0x7f63fc049b20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:31.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.008+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c570 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 rx=0x7f63fc0099d0 tx=0x7f63fc049b20).stop 2021-03-09T09:03:31.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:31.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:31.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:31.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c570 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:31.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:03:31.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:31.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:31.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:31.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:31.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:31.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:31.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:31.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b190 con 0x7f63e00070a0 2021-03-09T09:03:31.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:31.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001fdf0 con 0x7f6400145bd0 2021-03-09T09:03:31.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:31.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:03:31.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:31.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:31.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:31.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:31.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:31.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:31.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:03:31.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:31.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:31.021 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:31.009+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6441 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:31.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:31.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:31.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:31.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:31.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:31.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:31.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:31.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:31.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:03:31.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:03:31.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:31.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:31.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f820 con 0x7f63e005c7c0 2021-03-09T09:03:31.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:31.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:31.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.010+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00656f0 con 0x7f63e005c7c0 2021-03-09T09:03:31.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.011+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:03:31.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.011+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:31.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.011+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 rx=0x7f63f4005230 tx=0x7f63f40038a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:31.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.011+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:03:31.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.011+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:31.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404b270 con 0x7f63e005c7c0 2021-03-09T09:03:31.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.013+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:31.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.013+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:31.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.013+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:31.030 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:31.030 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:31.030 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:31.030 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:31.031 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:31.031 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:31.031 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:31.031 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:31.031 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:31.032 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:31.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404b270 con 0x7f63e005c7c0 2021-03-09T09:03:31.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.013+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:31.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.014+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:03:31.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.015+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:03:31.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:31.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:31.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:32.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:32.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:32.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:32.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:32.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:32.453+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:33.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:33.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:33.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:33.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:34.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:34.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:34.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:34.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:34.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:34.514+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:35.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:35.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:35.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:35.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0017d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0017d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 rx=0x7f63f4005230 tx=0x7f63f40038a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:36.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 rx=0x7f63f4005230 tx=0x7f63f40038a0).stop 2021-03-09T09:03:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0017d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:36.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:36.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078980 con 0x7f63e00070a0 2021-03-09T09:03:36.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e60 con 0x7f6400145bd0 2021-03-09T09:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022430 con 0x7f6400146050 2021-03-09T09:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:36.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:36.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:36.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:36.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:03:36.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:36.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.010+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:36.018 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6484 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:36.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:36.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:03:36.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:36.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:36.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a8f0 con 0x7f6400146050 2021-03-09T09:03:36.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:36.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:36.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001dd80 con 0x7f6400146050 2021-03-09T09:03:36.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.012+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:03:36.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.012+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:36.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.012+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:03:36.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.012+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:36.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.012+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc047d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:36.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:03:36.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.013+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:36.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.013+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:36.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.013+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:36.026 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:36.027 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:36.027 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:36.027 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:36.028 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:36.028 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:36.028 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:36.028 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:36.028 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:36.029 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:36.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f6400146050 2021-03-09T09:03:36.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.013+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:36.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.013+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:03:36.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.015+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:03:36.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:36.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:36.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:37.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:37.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:37.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:37.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:38.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:38.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:38.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:38.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:39.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:39.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:39.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:39.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:39.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:39.515+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:40.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:40.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:40.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:40.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:41.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:41.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:41.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc047d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:41.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 secure :-1 s=READY pgs=2468 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc047d30).stop 2021-03-09T09:03:41.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:41.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:41.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:41.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bd50 unknown :-1 s=CLOSED pgs=2468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:41.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:03:41.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00661e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:41.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:41.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:41.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:41.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:41.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:41.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:41.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:03:41.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:41.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b410 con 0x7f6400145bd0 2021-03-09T09:03:41.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:41.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078980 con 0x7f63e005c7c0 2021-03-09T09:03:41.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:41.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:41.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:41.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:41.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.011+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:41.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.012+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00661e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:41.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.012+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:03:41.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.012+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:41.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.012+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:41.041 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:41.012+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6524 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:41.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.013+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.013+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.013+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:41.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:41.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:41.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:41.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:03:41.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:03:41.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:41.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:41.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012900 con 0x7f63e005c7c0 2021-03-09T09:03:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f820 con 0x7f63e005c7c0 2021-03-09T09:03:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:03:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00661e0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 rx=0x7f63f400b160 tx=0x7f63f402a260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:03:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.014+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e005c7c0 2021-03-09T09:03:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.016+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.016+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.016+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:41.049 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:41.049 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:41.049 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:41.049 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:41.050 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:41.050 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:41.050 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:41.050 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:41.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005b60 con 0x7f63e005c7c0 2021-03-09T09:03:41.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.016+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:41.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.016+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:03:41.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f63e005c7c0 2021-03-09T09:03:41.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:41.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:41.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:42.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:42.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:42.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:42.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:42.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:42.454+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:43.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:43.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:43.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:43.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:44.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:44.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:44.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:44.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:44.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:44.516+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:45.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:45.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:45.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:45.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:46.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.012+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:46.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.012+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00661e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:46.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.012+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00661e0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 rx=0x7f63f400b160 tx=0x7f63f402a260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:46.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.012+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00661e0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 rx=0x7f63f400b160 tx=0x7f63f402a260).stop 2021-03-09T09:03:46.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.012+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:46.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.012+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:46.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00661e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:46.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.012+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00661e0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:46.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.012+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:03:46.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:46.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:46.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:46.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:46.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:46.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:46.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:46.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b180 con 0x7f63e00070a0 2021-03-09T09:03:46.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:46.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400145bd0 2021-03-09T09:03:46.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:46.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029440 con 0x7f6400146050 2021-03-09T09:03:46.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:46.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:46.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:46.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:46.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:46.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:46.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:03:46.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:46.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:46.023 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6551 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:46.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:46.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:46.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.013+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:46.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:46.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:46.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:46.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:46.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:46.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:03:46.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:03:46.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:46.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:46.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00258d0 con 0x7f6400146050 2021-03-09T09:03:46.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:46.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:46.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:03:46.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:03:46.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.014+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:46.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.015+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bef0 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63f005fed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:46.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.015+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:03:46.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.015+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:46.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006d90 con 0x7f6400146050 2021-03-09T09:03:46.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.017+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:46.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.017+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:46.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.017+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:46.030 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:46.031 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:46.031 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:46.031 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:46.031 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:46.032 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:46.032 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:46.032 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:46.033 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:46.033 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:46.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0054250 con 0x7f6400146050 2021-03-09T09:03:46.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.017+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:46.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.017+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:03:46.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T09:03:46.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:46.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:46.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:47.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:47.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:48.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:48.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:48.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:48.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:49.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:49.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:49.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:49.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:49.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:49.517+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:50.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:50.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:51.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:03:51.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:51.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bef0 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63f005fed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:51.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bef0 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63f005fed0).stop 2021-03-09T09:03:51.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:51.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:51.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:51.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bef0 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:51.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:03:51.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:51.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:51.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b410 con 0x7f63e00070a0 2021-03-09T09:03:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00111a0 con 0x7f6400145bd0 2021-03-09T09:03:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:51.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e005c7c0 2021-03-09T09:03:51.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:51.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:51.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:51.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:51.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:03:51.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:51.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.013+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:51.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:51.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:51.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:51.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:51.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:51.025 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6597 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:51.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:51.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:51.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:51.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:51.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:51.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:03:51.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:03:51.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:51.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:51.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f6f0 con 0x7f63e005c7c0 2021-03-09T09:03:51.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:51.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:51.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00258d0 con 0x7f63e005c7c0 2021-03-09T09:03:51.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:03:51.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.014+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:51.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.015+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:03:51.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.015+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c3e0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 rx=0x7f63e00356f0 tx=0x7f63f00553f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:51.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.015+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:51.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004c2c0 con 0x7f63e005c7c0 2021-03-09T09:03:51.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.016+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:51.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.016+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:51.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.016+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:51.031 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:51.031 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:51.032 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:51.032 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:51.032 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:51.032 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:51.032 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:51.033 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:51.033 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:51.034 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:51.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002d4a0 con 0x7f63e005c7c0 2021-03-09T09:03:51.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.016+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:51.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.016+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:03:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e005c7c0 2021-03-09T09:03:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:51.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:52.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:52.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:52.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:52.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:52.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:52.454+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:53.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:53.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:53.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:53.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:54.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:54.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:54.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:54.517+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:03:55.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:55.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:55.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:55.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:03:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:03:56.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c3e0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 rx=0x7f63e00356f0 tx=0x7f63f00553f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:03:56.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c3e0 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 rx=0x7f63e00356f0 tx=0x7f63f00553f0).stop 2021-03-09T09:03:56.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:56.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:03:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:03:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c3e0 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:03:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:03:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:56.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:03:56.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:56.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:03:56.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:03:56.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:03:56.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:56.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b3d0 con 0x7f63e00070a0 2021-03-09T09:03:56.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:56.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b180 con 0x7f6400145bd0 2021-03-09T09:03:56.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:03:56.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029730 con 0x7f6400146050 2021-03-09T09:03:56.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:03:56.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:03:56.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:03:56.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:56.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:03:56.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:03:56.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:03:56.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:03:56.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.014+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:03:56.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:03:56.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:03:56.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:03:56.026 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6638 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:03:56.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:03:56.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:56.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:56.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:03:56.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:03:56.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:03:56.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:03:56.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:03:56.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:56.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:03:56.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:03:56.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:03:56.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012900 con 0x7f6400146050 2021-03-09T09:03:56.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:03:56.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.015+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:56.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.016+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:03:56.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.016+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a420 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40035e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:03:56.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.016+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:03:56.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4069e30 con 0x7f6400146050 2021-03-09T09:03:56.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.017+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:03:56.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.017+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:03:56.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.017+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:03:56.033 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:03:56.033 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:03:56.034 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:56.034 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:03:56.034 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:03:56.034 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:03:56.035 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:03:56.035 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:03:56.035 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:03:56.036 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:03:56.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4017260 con 0x7f6400146050 2021-03-09T09:03:56.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.017+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:03:56.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.017+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:03:56.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:03:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:56.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:57.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:57.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:57.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:57.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:58.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:58.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:58.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:58.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:59.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:03:59.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:59.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:03:59.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:03:59.518+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:00.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:00.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:00.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:00.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a420 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40035e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a420 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40035e0).stop 2021-03-09T09:04:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:01.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007a420 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:01.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a420 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:01.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:04:01.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0008650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:01.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:01.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:01.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:01.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:01.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:01.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:01.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cf20 con 0x7f63e00070a0 2021-03-09T09:04:01.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:01.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:04:01.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:01.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b410 con 0x7f63e005c7c0 2021-03-09T09:04:01.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.015+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:01.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.016+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:01.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.016+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:01.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.016+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:01.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.016+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:01.026 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:01.016+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6657 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:01.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:01.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:04:01.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:01.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:01.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:01.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:01.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:01.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:01.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:01.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:01.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:01.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:01.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:04:01.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:04:01.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:01.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:01.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066550 con 0x7f63e005c7c0 2021-03-09T09:04:01.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:01.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:01.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:04:01.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:04:01.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.017+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:01.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.018+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0008650 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 rx=0x7f63f40627c0 tx=0x7f63f40093c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:01.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.018+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:04:01.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.018+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:01.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40623d0 con 0x7f63e005c7c0 2021-03-09T09:04:01.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.020+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:01.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.020+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:01.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.020+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:01.036 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63e005c7c0 2021-03-09T09:04:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.020+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.020+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:04:01.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.021+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4036b80 con 0x7f63e005c7c0 2021-03-09T09:04:01.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:01.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:01.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:02.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:02.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:02.456+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:03.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:03.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:03.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:03.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:04.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:04.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:04.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:04.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:04.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:04.519+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:05.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:05.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:05.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:05.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0008650 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 rx=0x7f63f40627c0 tx=0x7f63f40093c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0008650 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 rx=0x7f63f40627c0 tx=0x7f63f40093c0).stop 2021-03-09T09:04:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0008650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0008650 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:04:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00779e0 con 0x7f63e00070a0 2021-03-09T09:04:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b140 con 0x7f6400145bd0 2021-03-09T09:04:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400146050 2021-03-09T09:04:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:06.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:06.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:04:06.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:06.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:06.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:06.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:06.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:06.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:06.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:04:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:04:06.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:06.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:06.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:04:06.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:06.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:06.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066550 con 0x7f6400146050 2021-03-09T09:04:06.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:04:06.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:06.033 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:06.017+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6713 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c770 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f40341b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.018+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:06.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40295e0 con 0x7f6400146050 2021-03-09T09:04:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.019+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.019+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.019+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:06.035 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:06.037 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4062830 con 0x7f6400146050 2021-03-09T09:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.019+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.019+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:04:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.021+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4036b80 con 0x7f6400146050 2021-03-09T09:04:06.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:06.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:06.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:07.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:07.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:07.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:07.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:08.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:08.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:08.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:08.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:09.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:09.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:09.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:09.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:09.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:09.520+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:10.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:10.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:10.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:10.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c770 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f40341b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c770 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f40341b0).stop 2021-03-09T09:04:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007c770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007c770 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:04:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029730 con 0x7f63e00070a0 2021-03-09T09:04:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b3d0 con 0x7f6400145bd0 2021-03-09T09:04:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:11.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b180 con 0x7f63e005c7c0 2021-03-09T09:04:11.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:11.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:11.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:11.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:11.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.018+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:11.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.019+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.019+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:04:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.019+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.019+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.019+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.019+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.019+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:11.029 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:11.019+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6747 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:11.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:11.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:04:11.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:04:11.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:11.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:11.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019660 con 0x7f63e005c7c0 2021-03-09T09:04:11.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:11.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:11.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f6f0 con 0x7f63e005c7c0 2021-03-09T09:04:11.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:04:11.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.020+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.021+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:04:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.021+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.021+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cc60 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 rx=0x7f63e0001fa0 tx=0x7f63f004cf70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.022+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dd30 con 0x7f63e005c7c0 2021-03-09T09:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.022+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.022+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.022+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:11.036 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:11.036 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:11.036 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:11.037 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004dd30 con 0x7f63e005c7c0 2021-03-09T09:04:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.025+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f000e260 con 0x7f63e005c7c0 2021-03-09T09:04:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.026+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:04:11.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:11.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:11.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:12.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:12.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:12.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:12.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:12.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:12.457+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:13.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:13.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:13.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:13.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:14.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:14.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:14.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:14.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:14.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:14.521+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:15.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:15.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:15.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:15.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:16.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:16.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:16.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cc60 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 rx=0x7f63e0001fa0 tx=0x7f63f004cf70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:16.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cc60 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 rx=0x7f63e0001fa0 tx=0x7f63f004cf70).stop 2021-03-09T09:04:16.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:16.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:16.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007cc60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:16.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cc60 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:16.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:04:16.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:16.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:16.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:16.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:16.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:16.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:16.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:16.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d530 con 0x7f63e00070a0 2021-03-09T09:04:16.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:16.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:04:16.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:16.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001dd50 con 0x7f6400146050 2021-03-09T09:04:16.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:16.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:16.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:16.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:16.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.020+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:16.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.021+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:16.030 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:16.021+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6787 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:16.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.021+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:04:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.021+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.021+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.021+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.021+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:16.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.021+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:16.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:16.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:16.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:16.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:16.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:16.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:04:16.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:04:16.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:16.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:16.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001cd20 con 0x7f6400146050 2021-03-09T09:04:16.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:16.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:16.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:04:16.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:04:16.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.022+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:16.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.023+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010710 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f63f4051570 tx=0x7f63f40037c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:16.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.023+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:04:16.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.023+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:16.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.026+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40611a0 con 0x7f6400146050 2021-03-09T09:04:16.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.026+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:16.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.026+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:16.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.026+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:16.038 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:16.038 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:16.038 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:16.039 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:16.039 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:16.039 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:16.039 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:16.040 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034e20 con 0x7f6400146050 2021-03-09T09:04:16.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.028+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:16.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.028+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:04:16.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T09:04:16.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:16.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:16.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:17.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:17.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:17.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:17.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:18.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:18.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:18.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:18.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:19.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:19.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:19.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:19.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:19.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:19.522+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:20.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:20.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:20.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:20.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:21.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0010710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:21.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0010710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:21.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010710 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f63f4051570 tx=0x7f63f40037c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:21.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010710 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 rx=0x7f63f4051570 tx=0x7f63f40037c0).stop 2021-03-09T09:04:21.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:21.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:21.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0010710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:21.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010710 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:21.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:04:21.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:21.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:21.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:21.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:21.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:21.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:21.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:21.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001eb60 con 0x7f63e00070a0 2021-03-09T09:04:21.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:21.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b180 con 0x7f6400145bd0 2021-03-09T09:04:21.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:21.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029730 con 0x7f63e005c7c0 2021-03-09T09:04:21.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.021+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:21.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.022+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:21.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.022+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:21.030 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:21.022+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6824 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:21.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.022+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:21.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.022+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:21.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.022+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:21.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.022+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:04:21.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.022+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:21.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.022+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:21.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.023+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:21.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.023+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:21.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.023+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:21.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:21.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:21.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:21.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:21.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:21.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:04:21.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:04:21.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:21.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:21.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0030410 con 0x7f63e005c7c0 2021-03-09T09:04:21.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:21.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:21.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019660 con 0x7f63e005c7c0 2021-03-09T09:04:21.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:04:21.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.024+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:21.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7f63f003f920 tx=0x7f63f0013f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:21.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.025+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:04:21.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.025+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:21.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00554e0 con 0x7f63e005c7c0 2021-03-09T09:04:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.027+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.027+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.027+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:21.040 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e005c7c0 2021-03-09T09:04:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.029+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.029+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:04:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.030+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003f1d0 con 0x7f63e005c7c0 2021-03-09T09:04:21.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:21.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:21.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:22.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:22.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:22.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:22.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:22.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:22.458+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:23.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:23.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:23.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:24.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:24.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:24.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:24.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:24.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:24.523+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:25.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:25.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:25.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:25.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:26.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.022+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:26.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.022+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:26.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.022+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7f63f003f920 tx=0x7f63f0013f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:26.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.022+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 rx=0x7f63f003f920 tx=0x7f63f0013f70).stop 2021-03-09T09:04:26.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:26.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:26.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:26.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:26.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:04:26.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:26.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:26.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00193e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:26.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:26.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:26.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:26.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:26.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b410 con 0x7f63e00070a0 2021-03-09T09:04:26.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:26.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:04:26.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:26.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c820 con 0x7f6400146050 2021-03-09T09:04:26.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:26.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:26.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:26.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:26.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:26.032 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:26.023+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6862 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:26.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.024+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00193e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:26.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.024+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:04:26.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.024+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:26.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.024+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:26.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.024+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:26.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.024+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:26.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.024+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:26.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:26.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:26.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:26.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:26.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:26.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:04:26.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:04:26.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:26.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:26.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:04:26.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:26.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:26.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a8f0 con 0x7f6400146050 2021-03-09T09:04:26.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:04:26.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:26.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:04:26.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:26.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.026+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00193e0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 rx=0x7f63e004bf40 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:26.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:04:26.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.028+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:26.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.028+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:26.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.028+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:26.040 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:26.040 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:26.040 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:26.041 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:26.041 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:26.041 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:26.041 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:26.041 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:26.042 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:26.042 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:26.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:04:26.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.028+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:26.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.028+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:04:26.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:04:26.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:26.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:26.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:27.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:27.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:27.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:27.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:28.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:28.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:28.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:28.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:29.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:29.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:29.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:29.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:29.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:29.524+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:30.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:30.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:30.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:30.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:31.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.024+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00193e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:31.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.024+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00193e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:31.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.024+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00193e0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 rx=0x7f63e004bf40 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:31.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.024+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00193e0 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 rx=0x7f63e004bf40 tx=0x7f63fc009990).stop 2021-03-09T09:04:31.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:31.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:31.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00193e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:31.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00193e0 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:31.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:04:31.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:31.030 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6897 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:31.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:31.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:31.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:31.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:31.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:31.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:31.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e010 con 0x7f63e00070a0 2021-03-09T09:04:31.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:31.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029730 con 0x7f6400145bd0 2021-03-09T09:04:31.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:31.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ab40 con 0x7f63e005c7c0 2021-03-09T09:04:31.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:31.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:31.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:31.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:04:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.025+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:31.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:31.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:04:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:04:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:31.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:31.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:04:31.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:31.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:31.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0030410 con 0x7f63e005c7c0 2021-03-09T09:04:31.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:04:31.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:31.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005db30 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:31.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:04:31.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.026+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.030+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040870 con 0x7f63e005c7c0 2021-03-09T09:04:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.030+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.030+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.030+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:31.046 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:31.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.030+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055460 con 0x7f63e005c7c0 2021-03-09T09:04:31.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.030+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:31.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.030+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:04:31.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.030+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f63e005c7c0 2021-03-09T09:04:31.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:31.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:31.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:32.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:32.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:32.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:32.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:32.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:32.460+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:33.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:33.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:33.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:33.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:34.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:34.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:34.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:34.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:34.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:34.525+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:35.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:35.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:35.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:35.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:36.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e005db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:36.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e005db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:36.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005db30 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:36.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005db30 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006ca0).stop 2021-03-09T09:04:36.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:36.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:36.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e005db30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:36.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005db30 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:36.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:04:36.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:36.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:36.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:36.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:36.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:36.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:36.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:36.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:04:36.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:36.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00779e0 con 0x7f6400145bd0 2021-03-09T09:04:36.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:36.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f6400146050 2021-03-09T09:04:36.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:36.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:36.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:36.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:36.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:36.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.026+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:36.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.026+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:04:36.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.026+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:36.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.026+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:36.036 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:36.026+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6933 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:36.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.027+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:36.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.027+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:36.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.027+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:36.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:36.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:36.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:36.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:36.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:36.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:04:36.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:04:36.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:36.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:36.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f6b0 con 0x7f6400146050 2021-03-09T09:04:36.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:36.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:36.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:04:36.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:04:36.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.028+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:36.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.029+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e8d0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f63fc047d00 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:36.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.029+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:04:36.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.029+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:36.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:04:36.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.031+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:36.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.031+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:36.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.031+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:36.043 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:36.043 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:36.044 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:36.044 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:36.046 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc019350 con 0x7f6400146050 2021-03-09T09:04:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.032+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.032+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:04:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:04:36.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:36.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:36.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:37.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:37.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:37.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:37.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:38.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:38.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:38.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:38.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:39.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:39.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:39.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:39.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:39.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:39.526+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:40.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:40.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:40.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:40.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:41.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.026+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:41.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.026+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:41.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.026+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e8d0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f63fc047d00 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:41.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.026+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e8d0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 rx=0x7f63fc047d00 tx=0x7f63fc04a740).stop 2021-03-09T09:04:41.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:41.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:41.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:41.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e8d0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:41.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:04:41.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:41.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:41.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:41.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:41.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:41.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:41.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:41.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e00070a0 2021-03-09T09:04:41.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:41.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000da10 con 0x7f6400145bd0 2021-03-09T09:04:41.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:41.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077be0 con 0x7f63e005c7c0 2021-03-09T09:04:41.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:41.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:41.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:41.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:41.036 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 6975 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:41.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.027+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:41.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.029+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:41.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.029+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:04:41.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.029+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:41.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.029+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:41.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.030+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:41.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.030+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:41.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.030+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:41.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:41.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:41.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:41.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:41.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:41.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:04:41.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:04:41.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:41.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:41.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066550 con 0x7f63e005c7c0 2021-03-09T09:04:41.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:41.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f6b0 con 0x7f63e005c7c0 2021-03-09T09:04:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:04:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.031+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.033+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:04:41.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.033+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b610 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc02ce70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:41.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.033+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:41.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.034+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc032560 con 0x7f63e005c7c0 2021-03-09T09:04:41.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.034+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:41.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.034+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:41.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.034+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:41.045 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:41.045 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:41.045 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:41.045 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:41.045 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:41.047 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.037+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc076310 con 0x7f63e005c7c0 2021-03-09T09:04:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.037+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.037+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:04:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.037+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049700 con 0x7f63e005c7c0 2021-03-09T09:04:41.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:41.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:41.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:42.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:42.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:42.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:42.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:42.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:42.461+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:43.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:43.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:44.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:44.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:44.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:44.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:44.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:44.527+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:45.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:45.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:45.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:45.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:46.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:04:46.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:46.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b610 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc02ce70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:46.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b610 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc02ce70).stop 2021-03-09T09:04:46.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:46.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:46.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001b610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:46.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b610 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:46.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:04:46.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:46.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:46.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:46.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:46.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:46.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:46.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:46.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029730 con 0x7f63e00070a0 2021-03-09T09:04:46.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:46.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b410 con 0x7f6400145bd0 2021-03-09T09:04:46.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:46.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:04:46.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:46.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:46.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:46.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:46.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.028+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:46.038 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7002 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:46.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:46.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:04:46.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:46.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:46.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:46.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:46.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:46.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:46.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:46.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:46.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:46.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:46.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:04:46.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:04:46.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:46.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:46.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:04:46.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:46.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:46.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.029+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001cd20 con 0x7f6400146050 2021-03-09T09:04:46.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.030+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:04:46.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.030+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:46.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.030+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:04:46.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.030+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:46.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.030+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027ac0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7f63f401a840 tx=0x7f63f4003b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:46.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009340 con 0x7f6400146050 2021-03-09T09:04:46.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.031+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:46.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.031+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:46.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.031+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:46.046 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:46.046 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:46.046 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:46.047 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:46.047 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:46.047 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:46.047 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:46.047 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:46.048 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:46.048 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:46.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009340 con 0x7f6400146050 2021-03-09T09:04:46.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.031+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:46.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.032+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:04:46.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.037+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:04:46.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:46.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:46.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:47.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:47.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:47.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:47.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:48.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:48.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:48.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:48.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:49.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:49.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:49.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:49.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:49.528+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:50.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:50.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:50.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:50.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:51.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:04:51.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:51.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027ac0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7f63f401a840 tx=0x7f63f4003b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:51.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027ac0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 rx=0x7f63f401a840 tx=0x7f63f4003b40).stop 2021-03-09T09:04:51.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:51.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:51.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:51.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027ac0 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:51.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:04:51.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:51.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:51.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:51.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:51.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00779e0 con 0x7f63e00070a0 2021-03-09T09:04:51.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:51.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f6400145bd0 2021-03-09T09:04:51.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:51.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b2d0 con 0x7f63e005c7c0 2021-03-09T09:04:51.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:51.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:51.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.029+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:51.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.030+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:51.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.030+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:51.039 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:51.030+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7038 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:51.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.030+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:51.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.030+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:04:51.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.030+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:51.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.030+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:51.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:51.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:51.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:51.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:51.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:51.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:51.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:51.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:51.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:04:51.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:04:51.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:51.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:51.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014400 con 0x7f63e005c7c0 2021-03-09T09:04:51.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:51.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:51.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066550 con 0x7f63e005c7c0 2021-03-09T09:04:51.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:04:51.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.031+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:51.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.032+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:04:51.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.032+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 rx=0x7f63e000c6d0 tx=0x7f63fc052fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:51.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.032+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:51.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc076310 con 0x7f63e005c7c0 2021-03-09T09:04:51.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.033+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:51.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.033+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:51.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.033+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:51.049 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:51.049 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:51.049 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:51.050 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:51.050 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:51.050 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:51.050 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:51.051 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:51.051 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:51.051 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:51.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc076310 con 0x7f63e005c7c0 2021-03-09T09:04:51.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.033+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:51.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.034+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:04:51.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:04:51.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:51.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:51.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:52.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:52.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:52.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:52.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:52.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:52.462+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:53.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:53.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:53.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:53.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:54.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:54.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:54.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:54.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:54.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:54.529+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:04:55.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:55.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:55.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:55.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:56.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.030+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:04:56.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.030+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:04:56.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.030+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 rx=0x7f63e000c6d0 tx=0x7f63fc052fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:04:56.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.030+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 rx=0x7f63e000c6d0 tx=0x7f63fc052fb0).stop 2021-03-09T09:04:56.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.030+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:56.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.030+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:04:56.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.030+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:04:56.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.030+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:04:56.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.030+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:04:56.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:56.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:04:56.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:56.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:04:56.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:04:56.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:04:56.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:56.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:04:56.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:56.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f6400145bd0 2021-03-09T09:04:56.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:04:56.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029f40 con 0x7f6400146050 2021-03-09T09:04:56.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:04:56.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:04:56.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:56.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:04:56.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:04:56.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:04:56.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:04:56.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:04:56.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:04:56.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:04:56.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:04:56.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:04:56.043 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:04:56.031+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7073 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:04:56.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:04:56.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:56.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:56.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:04:56.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:04:56.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:04:56.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:04:56.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:04:56.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:56.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d60 con 0x7f6400146050 2021-03-09T09:04:56.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:04:56.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:04:56.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:04:56.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:04:56.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.032+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:56.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.033+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:04:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.033+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:04:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.033+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015e10 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 rx=0x7f63f4003b10 tx=0x7f63f40623d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:04:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.034+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4062600 con 0x7f6400146050 2021-03-09T09:04:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.034+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:04:56.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.034+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:04:56.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.034+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:04:56.051 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:04:56.051 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:04:56.052 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:56.052 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:04:56.052 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:04:56.052 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:04:56.053 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:04:56.053 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:04:56.053 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:04:56.054 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:04:56.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.034+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f401a480 con 0x7f6400146050 2021-03-09T09:04:56.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.034+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:04:56.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.034+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:04:56.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:04:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:56.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:57.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:57.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:58.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:58.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:58.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:58.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:59.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:04:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:59.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:04:59.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:04:59.530+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:00.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:00.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:00.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:00.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:01.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015e10 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 rx=0x7f63f4003b10 tx=0x7f63f40623d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:01.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015e10 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 rx=0x7f63f4003b10 tx=0x7f63f40623d0).stop 2021-03-09T09:05:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015e10 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:01.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:05:01.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:01.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:01.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:01.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:01.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00793d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:01.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:01.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:01.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:05:01.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f6400145bd0 2021-03-09T09:05:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ac00 con 0x7f63e005c7c0 2021-03-09T09:05:01.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:01.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.032+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00793d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.033+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:05:01.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.033+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:01.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.033+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:01.042 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:01.033+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7119 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:01.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.033+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:01.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.033+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:01.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.033+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:01.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:01.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:01.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:01.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:01.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:01.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:05:01.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:05:01.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:01.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:01.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001cd70 con 0x7f63e005c7c0 2021-03-09T09:05:01.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:01.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:01.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:05:01.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:05:01.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.034+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:01.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00793d0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 rx=0x7f63f001c000 tx=0x7f63f004de10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:01.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.036+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:05:01.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.036+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:01.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003f1d0 con 0x7f63e005c7c0 2021-03-09T09:05:01.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.036+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:01.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.036+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:01.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.036+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:01.050 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:01.051 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:01.051 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:01.051 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:01.051 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:01.051 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:01.052 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.037+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006170 con 0x7f63e005c7c0 2021-03-09T09:05:01.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.037+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:01.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.037+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:05:01.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f63e005c7c0 2021-03-09T09:05:01.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:01.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:01.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:02.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:02.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:02.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:02.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:02.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:02.464+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:03.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:03.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:03.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:03.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:04.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:04.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:04.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:04.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:04.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:04.531+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:05.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:05.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:05.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:05.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:06.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00793d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00793d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00793d0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 rx=0x7f63f001c000 tx=0x7f63f004de10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:06.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00793d0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 rx=0x7f63f001c000 tx=0x7f63f004de10).stop 2021-03-09T09:05:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:06.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:06.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00793d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:06.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00793d0 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:06.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:05:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:06.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:06.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:06.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:06.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:05:06.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:06.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400145bd0 2021-03-09T09:05:06.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:06.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001fd50 con 0x7f6400146050 2021-03-09T09:05:06.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.033+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:06.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.034+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.034+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:06.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.034+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:06.044 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:06.034+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7157 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:06.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.034+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.034+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:05:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.034+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.034+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:06.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:06.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:06.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:05:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:05:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:06.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001d60 con 0x7f6400146050 2021-03-09T09:05:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002d60 con 0x7f6400146050 2021-03-09T09:05:06.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:05:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.035+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.036+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:05:06.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.036+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.036+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f40036b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.037+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404b270 con 0x7f6400146050 2021-03-09T09:05:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.037+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.037+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.037+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:06.056 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.037+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404b3d0 con 0x7f6400146050 2021-03-09T09:05:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.037+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.037+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:05:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4036260 con 0x7f6400146050 2021-03-09T09:05:06.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:06.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:06.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:07.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:07.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:07.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:07.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:08.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:08.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:08.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:08.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:09.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:09.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:09.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:09.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:09.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:09.532+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:10.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:10.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:10.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:10.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:05:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f40036b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:11.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f40036b0).stop 2021-03-09T09:05:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:11.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:05:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:11.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e210 con 0x7f63e00070a0 2021-03-09T09:05:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:11.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:05:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f63e005c7c0 2021-03-09T09:05:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:11.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:05:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:11.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.035+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:11.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:11.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:11.046 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7190 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:11.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:11.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:11.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:11.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:11.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:11.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:05:11.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:05:11.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:11.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:11.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000dfc0 con 0x7f63e005c7c0 2021-03-09T09:05:11.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:11.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:11.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014400 con 0x7f63e005c7c0 2021-03-09T09:05:11.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:05:11.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.036+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:11.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.037+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:05:11.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.037+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:11.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.037+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc052260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:11.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e005c7c0 2021-03-09T09:05:11.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.039+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:11.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.039+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:11.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.039+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:11.053 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:11.053 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:11.054 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:11.054 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:11.054 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:11.054 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:11.055 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:11.055 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:11.055 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:11.055 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:11.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:05:11.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.039+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:11.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.039+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:05:11.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f63e005c7c0 2021-03-09T09:05:11.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:11.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:11.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:12.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:12.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:12.465+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:13.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:13.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:13.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:13.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:14.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:14.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:14.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:14.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:14.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:14.533+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:15.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:15.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:16.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:05:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc052260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc052260).stop 2021-03-09T09:05:16.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:16.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:16.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:16.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd20 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:16.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:05:16.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:16.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:16.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:16.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:16.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:16.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:16.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:16.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:05:16.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:16.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024610 con 0x7f6400145bd0 2021-03-09T09:05:16.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:16.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028710 con 0x7f6400146050 2021-03-09T09:05:16.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:16.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:16.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:16.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.036+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:16.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:16.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:16.046 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7230 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:16.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:05:16.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:16.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:16.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:16.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:16.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:16.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:16.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:16.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:16.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:16.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:16.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:05:16.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.037+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:05:16.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:16.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:16.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e210 con 0x7f6400146050 2021-03-09T09:05:16.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:16.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:16.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001d60 con 0x7f6400146050 2021-03-09T09:05:16.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:05:16.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:16.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013ef0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7f63f4046cc0 tx=0x7f63f40621f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:16.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:05:16.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.038+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:16.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4061720 con 0x7f6400146050 2021-03-09T09:05:16.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.039+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:16.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.039+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:16.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.039+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:16.054 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:16.054 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:16.054 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:16.055 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:16.055 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:16.055 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:16.055 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:16.055 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:16.056 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:16.056 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:16.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4061720 con 0x7f6400146050 2021-03-09T09:05:16.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.040+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:16.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.040+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:05:16.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T09:05:16.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:16.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:16.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:17.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:17.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:17.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:17.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:18.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:18.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:18.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:18.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:19.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:19.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:19.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:19.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:19.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:19.534+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:20.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:20.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:20.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:20.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0013ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0013ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013ef0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7f63f4046cc0 tx=0x7f63f40621f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:21.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013ef0 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 rx=0x7f63f4046cc0 tx=0x7f63f40621f0).stop 2021-03-09T09:05:21.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:21.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:21.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0013ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:21.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013ef0 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:21.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:05:21.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:21.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:21.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:21.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:21.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:21.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:21.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:21.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065730 con 0x7f63e00070a0 2021-03-09T09:05:21.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:21.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013390 con 0x7f6400145bd0 2021-03-09T09:05:21.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:21.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010130 con 0x7f63e005c7c0 2021-03-09T09:05:21.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:21.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:21.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:21.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:21.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:21.051 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:21.038+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7268 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:21.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.039+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:21.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.039+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:05:21.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.039+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:21.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.039+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:21.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.039+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:21.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.039+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:21.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.039+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:21.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:21.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:21.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:21.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:21.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:05:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:05:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00225c0 con 0x7f63e005c7c0 2021-03-09T09:05:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000dfc0 con 0x7f63e005c7c0 2021-03-09T09:05:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:05:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.040+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.042+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:05:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.042+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:21.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c770 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7f63e005da60 tx=0x7f63fc00ff50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:21.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.043+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04f000 con 0x7f63e005c7c0 2021-03-09T09:05:21.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.043+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:21.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.043+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:21.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.043+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:21.068 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04f000 con 0x7f63e005c7c0 2021-03-09T09:05:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.045+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.045+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:05:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.046+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f63e005c7c0 2021-03-09T09:05:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:21.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:22.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:22.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:22.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:22.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:22.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:22.466+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:23.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:23.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:23.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:24.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:24.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:24.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:24.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:24.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:24.535+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:25.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:25.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:25.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:25.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:26.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:26.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:26.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c770 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7f63e005da60 tx=0x7f63fc00ff50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:26.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c770 secure :-1 s=READY pgs=2764 cs=0 l=1 rev1=1 rx=0x7f63e005da60 tx=0x7f63fc00ff50).stop 2021-03-09T09:05:26.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:26.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:26.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:26.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c770 unknown :-1 s=CLOSED pgs=2764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:26.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:05:26.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:26.044 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7311 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:26.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:26.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:05:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065730 con 0x7f6400145bd0 2021-03-09T09:05:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009270 con 0x7f6400146050 2021-03-09T09:05:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.039+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:05:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.040+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:05:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:05:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c470 con 0x7f6400146050 2021-03-09T09:05:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:26.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:26.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e210 con 0x7f6400146050 2021-03-09T09:05:26.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:05:26.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:26.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:05:26.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f63f402a210 tx=0x7f63f4017c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.042+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.043+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b4f0 con 0x7f6400146050 2021-03-09T09:05:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.043+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.043+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.043+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:26.059 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:26.061 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:26.061 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:26.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.047+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b4f0 con 0x7f6400146050 2021-03-09T09:05:26.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.047+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:26.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.048+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:05:26.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404b270 con 0x7f6400146050 2021-03-09T09:05:26.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:26.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:26.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:27.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:27.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:27.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:27.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:28.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:28.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:29.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:29.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:29.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:29.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:29.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:29.536+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:30.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:30.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:30.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:30.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0011b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0011b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:31.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f63f402a210 tx=0x7f63f4017c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 rx=0x7f63f402a210 tx=0x7f63f4017c30).stop 2021-03-09T09:05:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0011b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:31.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:05:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:31.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:31.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:31.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:31.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:31.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e0d0 con 0x7f63e00070a0 2021-03-09T09:05:31.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:31.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013390 con 0x7f6400145bd0 2021-03-09T09:05:31.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:31.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:05:31.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:31.049 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7344 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:31.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:05:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:31.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:31.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:31.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:31.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:31.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.041+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:31.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:31.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:05:31.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:05:31.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e210 con 0x7f63e005c7c0 2021-03-09T09:05:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00225c0 con 0x7f63e005c7c0 2021-03-09T09:05:31.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:05:31.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:31.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:05:31.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023390 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f63e00279d0 tx=0x7f63fc052340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:31.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.042+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:31.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065ca0 con 0x7f63e005c7c0 2021-03-09T09:05:31.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.044+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:31.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.044+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:31.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.044+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:31.057 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:31.058 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:31.058 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:31.058 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:31.060 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065a70 con 0x7f63e005c7c0 2021-03-09T09:05:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.044+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.044+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:05:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f63e005c7c0 2021-03-09T09:05:31.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:31.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:32.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:32.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:32.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:32.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:32.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:32.468+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:33.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:33.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:33.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:33.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:34.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:34.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:34.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:34.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:34.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:34.537+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:35.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:35.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:35.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:35.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:36.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:36.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023390 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f63e00279d0 tx=0x7f63fc052340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023390 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 rx=0x7f63e00279d0 tx=0x7f63fc052340).stop 2021-03-09T09:05:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:36.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023390 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:05:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:36.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:36.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:36.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065730 con 0x7f63e00070a0 2021-03-09T09:05:36.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:36.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ce40 con 0x7f6400145bd0 2021-03-09T09:05:36.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:36.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:05:36.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:36.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:36.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:36.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:36.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:36.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:36.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:05:36.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:36.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:36.052 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:36.042+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7385 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:36.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:36.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:36.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:36.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:36.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:36.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:36.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:36.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:36.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:05:36.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:05:36.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:36.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:36.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0009840 con 0x7f6400146050 2021-03-09T09:05:36.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001cd70 con 0x7f6400146050 2021-03-09T09:05:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:05:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.043+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:36.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 rx=0x7f63f0065810 tx=0x7f63f0065490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:05:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000e6c0 con 0x7f6400146050 2021-03-09T09:05:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:36.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:36.059 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:36.059 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:36.059 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:36.061 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:36.061 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:36.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f6400146050 2021-03-09T09:05:36.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:36.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.044+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:05:36.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.046+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:05:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:36.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:37.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:37.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:37.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:37.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:38.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:38.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:38.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:38.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:39.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:39.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:39.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:39.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:39.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:39.538+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:40.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:40.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:40.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:40.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0011b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:05:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0011b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:41.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 rx=0x7f63f0065810 tx=0x7f63f0065490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 rx=0x7f63f0065810 tx=0x7f63f0065490).stop 2021-03-09T09:05:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:41.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0011b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0011b10 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:41.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:05:41.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:41.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:41.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:41.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:41.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:41.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:41.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:41.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63e00070a0 2021-03-09T09:05:41.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:41.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ee60 con 0x7f6400145bd0 2021-03-09T09:05:41.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:41.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e0d0 con 0x7f63e005c7c0 2021-03-09T09:05:41.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:41.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:41.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:41.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:41.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:41.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.044+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:41.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.044+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:05:41.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.044+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:41.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.044+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:41.056 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:41.044+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7420 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:41.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.044+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:41.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.044+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:41.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.044+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:41.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:41.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:41.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:41.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:41.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:41.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:05:41.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:05:41.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:41.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:41.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017160 con 0x7f63e005c7c0 2021-03-09T09:05:41.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:41.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:41.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c470 con 0x7f63e005c7c0 2021-03-09T09:05:41.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:05:41.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.045+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:41.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.046+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:05:41.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.046+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:41.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.046+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f63f400b010 tx=0x7f63f4022990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:41.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.047+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e005c7c0 2021-03-09T09:05:41.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.047+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:41.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.047+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.047+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:41.068 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.047+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4069780 con 0x7f63e005c7c0 2021-03-09T09:05:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.047+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.047+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:05:41.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.047+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:05:41.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:41.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:42.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:42.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:42.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:42.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:42.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:42.468+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:43.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:43.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:43.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:43.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:44.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:44.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:44.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:44.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:44.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:44.539+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:45.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:45.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:45.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:45.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:46.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:05:46.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:46.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f63f400b010 tx=0x7f63f4022990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:46.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 rx=0x7f63f400b010 tx=0x7f63f4022990).stop 2021-03-09T09:05:46.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:46.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:46.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:46.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:46.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:05:46.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:46.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:46.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:46.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:46.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:46.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:46.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ce40 con 0x7f63e00070a0 2021-03-09T09:05:46.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:46.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d1f0 con 0x7f6400145bd0 2021-03-09T09:05:46.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:46.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017020 con 0x7f6400146050 2021-03-09T09:05:46.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:46.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:46.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:46.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:46.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:46.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:46.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:05:46.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:46.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:46.063 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:46.045+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7458 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:46.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.046+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:46.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.046+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:46.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.046+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:46.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:46.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:46.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:05:46.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:05:46.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:46.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:46.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0030410 con 0x7f6400146050 2021-03-09T09:05:46.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:46.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:46.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e210 con 0x7f6400146050 2021-03-09T09:05:46.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:05:46.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.047+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:46.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.048+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d720 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc052f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:46.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.048+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:05:46.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.048+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:46.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc076130 con 0x7f6400146050 2021-03-09T09:05:46.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.049+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.049+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.049+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:46.080 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:46.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f6400146050 2021-03-09T09:05:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.050+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.050+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:05:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:05:46.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:46.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:46.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:47.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:47.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:47.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:47.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:48.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:48.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:48.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:48.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:49.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:49.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:49.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:49.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:49.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:49.540+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:50.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:50.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:50.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:50.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:51.049 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7491 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:51.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:05:51.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:51.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d720 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc052f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:51.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d720 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc052f30).stop 2021-03-09T09:05:51.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:51.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:51.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006d720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:51.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d720 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:51.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:05:51.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:51.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:51.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:51.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:51.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:51.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:51.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:51.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079fe0 con 0x7f63e00070a0 2021-03-09T09:05:51.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:51.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019190 con 0x7f6400145bd0 2021-03-09T09:05:51.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:51.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:05:51.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:51.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:51.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.048+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:05:51.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.048+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:51.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.048+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:51.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.048+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:51.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.048+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:51.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.048+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:51.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:51.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:51.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:51.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:51.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:51.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:05:51.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:05:51.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:51.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:51.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63e005c7c0 2021-03-09T09:05:51.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:51.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:51.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0030410 con 0x7f63e005c7c0 2021-03-09T09:05:51.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:05:51.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:51.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:05:51.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:51.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.049+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ae80 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc06b430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:51.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06b0f0 con 0x7f63e005c7c0 2021-03-09T09:05:51.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.050+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:51.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.050+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:51.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.050+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:51.065 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:51.065 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:51.067 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:51.067 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:51.067 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:51.067 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:51.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06b0f0 con 0x7f63e005c7c0 2021-03-09T09:05:51.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.052+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:51.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.052+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:05:51.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e005c7c0 2021-03-09T09:05:51.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:51.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:51.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:52.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:52.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:52.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:52.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:52.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:52.470+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:53.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:53.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:53.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:53.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:54.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:54.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:54.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:54.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:54.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:54.541+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:05:55.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:55.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:55.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:55.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:05:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:05:56.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ae80 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc06b430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:05:56.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ae80 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc06b430).stop 2021-03-09T09:05:56.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:56.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:05:56.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006ae80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:05:56.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ae80 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:05:56.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:05:56.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:56.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:05:56.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:56.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:05:56.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:05:56.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:05:56.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:56.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ce40 con 0x7f63e00070a0 2021-03-09T09:05:56.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:56.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f6400145bd0 2021-03-09T09:05:56.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:05:56.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079a70 con 0x7f6400146050 2021-03-09T09:05:56.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:05:56.056 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7523 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:05:56.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:05:56.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:56.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:05:56.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:05:56.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.048+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:05:56.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.049+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:05:56.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.049+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:05:56.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.049+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:05:56.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.049+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:05:56.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.049+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:05:56.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.049+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:05:56.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:05:56.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:56.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:56.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:05:56.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:05:56.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:05:56.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:05:56.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:05:56.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:56.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e210 con 0x7f6400146050 2021-03-09T09:05:56.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:05:56.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:05:56.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:05:56.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:05:56.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.050+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:56.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.051+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:05:56.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.051+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:05:56.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.051+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc065fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:05:56.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc077360 con 0x7f6400146050 2021-03-09T09:05:56.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.053+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:05:56.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.053+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:05:56.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.053+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:05:56.065 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:05:56.065 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:05:56.065 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:56.066 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:05:56.066 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:05:56.066 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:05:56.066 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:05:56.067 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:05:56.067 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:05:56.067 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:05:56.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f6400146050 2021-03-09T09:05:56.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.053+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:05:56.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.053+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:05:56.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T09:05:56.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:56.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:56.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:57.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:57.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:57.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:57.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:58.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:58.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:58.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:58.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:59.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:59.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:05:59.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:59.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:05:59.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:05:59.542+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:00.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:00.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:00.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:00.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:01.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:01.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc065fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc065fb0).stop 2021-03-09T09:06:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:01.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:01.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:01.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:01.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:06:01.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:01.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:01.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:01.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:01.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00250e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:01.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001fd50 con 0x7f63e00070a0 2021-03-09T09:06:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:01.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:06:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019190 con 0x7f63e005c7c0 2021-03-09T09:06:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:01.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:01.057 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7565 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:01.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.049+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00250e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:01.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:06:01.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:01.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:01.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:01.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:01.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:06:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:06:01.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:01.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:01.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f820 con 0x7f63e005c7c0 2021-03-09T09:06:01.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:01.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:01.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0009840 con 0x7f63e005c7c0 2021-03-09T09:06:01.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:06:01.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.050+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:01.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:06:01.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00250e0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f0065330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:01.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:01.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0048400 con 0x7f63e005c7c0 2021-03-09T09:06:01.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:01.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:01.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:01.067 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:01.067 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:01.067 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:01.067 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:01.068 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:01.068 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:01.068 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:01.068 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:01.068 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:01.069 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:01.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f63e005c7c0 2021-03-09T09:06:01.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:01.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.051+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:06:01.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f63e005c7c0 2021-03-09T09:06:01.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:01.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:01.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:02.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:02.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:02.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:02.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:02.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:02.472+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:03.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:03.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:03.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:03.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:04.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:04.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:04.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:04.543+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:05.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:05.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:05.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:05.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:06.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00250e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00250e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00250e0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f0065330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:06.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00250e0 secure :-1 s=READY pgs=2866 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f0065330).stop 2021-03-09T09:06:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00250e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:06.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00250e0 unknown :-1 s=CLOSED pgs=2866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:06:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:06.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:06.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:06.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ce40 con 0x7f63e00070a0 2021-03-09T09:06:06.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:06.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065730 con 0x7f6400145bd0 2021-03-09T09:06:06.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:06.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001910 con 0x7f6400146050 2021-03-09T09:06:06.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:06.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:06.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:06.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:06.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:06.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:06.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:06:06.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:06.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.050+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:06.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:06.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:06.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:06.063 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7608 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:06.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:06.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:06.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:06.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:06.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:06.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:06:06.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:06:06.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:06.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:06.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ccd0 con 0x7f6400146050 2021-03-09T09:06:06.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:06.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:06.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:06:06.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.052+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:06:06.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.052+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:06.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.052+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:06:06.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.053+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:06.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f001a430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:06.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00485e0 con 0x7f6400146050 2021-03-09T09:06:06.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.054+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:06.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.054+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:06.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.054+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:06.074 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:06.074 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:06.075 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:06.075 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:06.076 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:06.076 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:06.077 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:06.077 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:06.077 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:06.078 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:06.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.056+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:06:06.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.056+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:06.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.056+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:06:06.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f6400146050 2021-03-09T09:06:06.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:06.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:06.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:07.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:07.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:07.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:07.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:08.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:08.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:08.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:08.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:09.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:09.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:09.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:09.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:09.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:09.544+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:10.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:10.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:10.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:10.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:11.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:06:11.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:11.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f001a430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:11.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f001a430).stop 2021-03-09T09:06:11.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:11.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:11.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000b9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:11.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:06:11.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:11.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:11.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:11.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:11.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:11.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:11.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006cb90 con 0x7f63e00070a0 2021-03-09T09:06:11.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:11.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400145bd0 2021-03-09T09:06:11.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:11.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00157c0 con 0x7f63e005c7c0 2021-03-09T09:06:11.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:11.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:11.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:06:11.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:11.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:11.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:11.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:11.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:11.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:11.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:11.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:11.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.052+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:11.070 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7646 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:11.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:11.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:11.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:11.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:06:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:06:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00786a0 con 0x7f63e005c7c0 2021-03-09T09:06:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:11.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:11.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017160 con 0x7f63e005c7c0 2021-03-09T09:06:11.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:06:11.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.053+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:11.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.054+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:06:11.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.054+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:11.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.054+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002bb10 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f400b050).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:11.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009350 con 0x7f63e005c7c0 2021-03-09T09:06:11.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.055+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:11.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.055+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:11.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.055+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:11.086 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:11.086 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:11.087 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:11.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40035e0 con 0x7f63e005c7c0 2021-03-09T09:06:11.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.055+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:11.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.055+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:06:11.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.056+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4036b70 con 0x7f63e005c7c0 2021-03-09T09:06:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:11.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:12.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:12.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:12.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:12.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:12.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:12.473+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:13.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:13.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:13.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:13.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:14.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:14.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:14.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:14.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:14.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:14.545+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:15.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:15.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:16.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:16.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:16.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002bb10 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f400b050).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:16.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002bb10 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f400b050).stop 2021-03-09T09:06:16.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:16.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:16.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:16.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002bb10 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:16.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:06:16.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:16.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:16.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:16.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:16.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:16.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:16.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:16.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019190 con 0x7f63e00070a0 2021-03-09T09:06:16.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:16.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:06:16.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:16.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ce40 con 0x7f6400146050 2021-03-09T09:06:16.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:16.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:16.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:16.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.054+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:16.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:16.062 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7686 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:16.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:16.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:06:16.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:16.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:16.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:16.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:06:16.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000bee0 con 0x7f6400146050 2021-03-09T09:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:16.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e210 con 0x7f6400146050 2021-03-09T09:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:16.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:06:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.056+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc052fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T09:06:16.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.057+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:16.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.057+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:16.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.057+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:16.070 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:16.070 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:16.071 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:16.071 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:16.071 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:16.071 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:16.072 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:16.072 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:16.072 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:16.072 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:16.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f6400146050 2021-03-09T09:06:16.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.057+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:16.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.057+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:06:16.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:06:16.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:16.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:16.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:17.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:17.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:18.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:18.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:18.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:18.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:19.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:19.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:19.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:19.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:19.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:19.546+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:20.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:20.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:20.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:20.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:21.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:21.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:21.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc052fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:21.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc052fa0).stop 2021-03-09T09:06:21.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:21.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:21.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000b9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000b9f0 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:06:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:21.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:21.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e00070a0 2021-03-09T09:06:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00157c0 con 0x7f6400145bd0 2021-03-09T09:06:21.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cbe0 con 0x7f63e005c7c0 2021-03-09T09:06:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.055+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.056+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:21.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.056+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:21.063 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:21.056+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7712 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.057+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.057+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:06:21.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.057+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.057+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.057+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.057+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.057+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:21.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.057+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.057+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:21.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:06:21.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:06:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018c50 con 0x7f63e005c7c0 2021-03-09T09:06:21.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ccd0 con 0x7f63e005c7c0 2021-03-09T09:06:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:06:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.058+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:21.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023c90 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:21.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.059+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:06:21.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.059+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:21.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001a290 con 0x7f63e005c7c0 2021-03-09T09:06:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.060+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.060+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.060+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:21.072 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:21.074 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:21.074 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:21.074 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:21.074 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:21.075 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:21.075 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:21.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.062+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001a290 con 0x7f63e005c7c0 2021-03-09T09:06:21.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.062+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:21.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.062+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:06:21.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f63e005c7c0 2021-03-09T09:06:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:21.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:22.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:22.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:22.474+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:23.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:23.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:23.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:23.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:24.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:24.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:24.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:24.547+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:25.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:25.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:25.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:25.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.057+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.057+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:26.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.057+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023c90 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:26.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.057+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023c90 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f0013b80).stop 2021-03-09T09:06:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023c90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023c90 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:26.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:06:26.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:26.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:26.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:26.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:26.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:26.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:26.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:26.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010330 con 0x7f63e00070a0 2021-03-09T09:06:26.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:26.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018ea0 con 0x7f6400145bd0 2021-03-09T09:06:26.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:26.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:06:26.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:26.064 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7758 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:26.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:26.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:26.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:26.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.058+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:26.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:26.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.059+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:06:26.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.059+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:26.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.059+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:26.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.060+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:26.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.060+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:26.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.060+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:26.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:26.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:26.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:26.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:26.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:26.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:06:26.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:06:26.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:26.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:26.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00656e0 con 0x7f6400146050 2021-03-09T09:06:26.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:26.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:26.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000bee0 con 0x7f6400146050 2021-03-09T09:06:26.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:06:26.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:26.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c070 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc0761a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:26.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:06:26.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.061+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:26.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:06:26.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.066+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:26.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.066+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:26.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.066+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:26.073 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:26.073 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:26.074 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:26.074 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:26.074 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:26.074 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:26.074 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:26.075 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:26.075 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:26.075 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:26.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f6400146050 2021-03-09T09:06:26.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.066+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:26.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.066+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:06:26.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:06:26.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:26.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:26.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:27.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:27.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:27.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:27.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:28.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:28.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:28.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:28.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:29.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:29.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:29.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:29.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:29.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:29.548+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:30.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:30.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:30.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:30.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c070 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc0761a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c070 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc0761a0).stop 2021-03-09T09:06:31.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:31.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:31.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:31.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c070 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:31.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:06:31.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:31.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:31.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00298d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025cb0 con 0x7f63e00070a0 2021-03-09T09:06:31.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:31.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400145bd0 2021-03-09T09:06:31.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:31.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b8b0 con 0x7f63e005c7c0 2021-03-09T09:06:31.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:31.066 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7806 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:31.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:31.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:31.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:31.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:31.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.060+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00298d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:31.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.060+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:06:31.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.060+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:31.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.060+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:31.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.061+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:31.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.061+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:31.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.061+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:31.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.061+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:31.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:31.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:31.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:31.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:31.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:06:31.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:06:31.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:31.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:31.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0023050 con 0x7f63e005c7c0 2021-03-09T09:06:31.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:31.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:31.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00656e0 con 0x7f63e005c7c0 2021-03-09T09:06:31.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:06:31.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:31.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:06:31.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:31.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.062+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00298d0 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 rx=0x7f63e00792d0 tx=0x7f63fc02c290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:31.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c600 con 0x7f63e005c7c0 2021-03-09T09:06:31.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.063+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:31.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.063+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:31.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.063+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:31.076 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:31.076 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:31.076 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:31.077 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:31.077 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:31.077 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:31.077 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:31.077 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:31.078 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:31.078 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:31.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc064950 con 0x7f63e005c7c0 2021-03-09T09:06:31.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.065+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:31.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.065+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:06:31.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f63e005c7c0 2021-03-09T09:06:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:31.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:32.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:32.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:32.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:32.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:32.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:32.476+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:33.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:33.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:33.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:33.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:34.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:34.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:34.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:34.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:34.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:34.549+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:35.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:35.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:35.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:35.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:36.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.060+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00298d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:36.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.060+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00298d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:36.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.060+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00298d0 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 rx=0x7f63e00792d0 tx=0x7f63fc02c290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:36.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.060+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00298d0 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 rx=0x7f63e00792d0 tx=0x7f63fc02c290).stop 2021-03-09T09:06:36.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.060+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:36.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.060+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:36.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00298d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:36.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.060+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00298d0 unknown :-1 s=CLOSED pgs=2958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:36.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:06:36.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:36.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:36.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:36.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:36.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:36.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:36.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:36.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e460 con 0x7f63e00070a0 2021-03-09T09:06:36.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027670 con 0x7f6400145bd0 2021-03-09T09:06:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ce40 con 0x7f6400146050 2021-03-09T09:06:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:36.068 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7838 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:36.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:36.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:36.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:06:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.061+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:36.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:36.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:36.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:36.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:06:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:06:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012650 con 0x7f6400146050 2021-03-09T09:06:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.062+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018c50 con 0x7f6400146050 2021-03-09T09:06:36.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.063+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:06:36.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.063+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:36.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.063+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:06:36.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.063+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:36.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.063+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012160 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000eef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:36.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T09:06:36.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.067+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:36.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.067+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:36.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.067+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:36.076 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:36.076 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:36.076 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:36.076 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:36.076 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:36.077 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:36.077 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:36.077 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:36.077 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:36.077 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:36.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d7a0 con 0x7f6400146050 2021-03-09T09:06:36.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.067+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:36.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.068+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:06:36.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:06:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:36.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:36.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:37.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:37.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:37.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:37.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:38.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:38.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:38.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:38.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:39.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:39.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:39.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:39.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:39.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:39.549+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:40.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:40.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:40.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:40.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:41.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:41.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:41.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012160 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000eef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:41.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012160 secure :-1 s=READY pgs=2970 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000eef0).stop 2021-03-09T09:06:41.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:41.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:41.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:41.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012160 unknown :-1 s=CLOSED pgs=2970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:41.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:06:41.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:41.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:06:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077d50 con 0x7f6400145bd0 2021-03-09T09:06:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065730 con 0x7f63e005c7c0 2021-03-09T09:06:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.060+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:41.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:41.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.061+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:06:41.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.061+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:41.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.061+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:41.070 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:41.061+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7873 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:41.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.061+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:41.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.061+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:41.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.061+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:41.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:41.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:06:41.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:06:41.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:41.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:41.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000bee0 con 0x7f63e005c7c0 2021-03-09T09:06:41.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:41.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:41.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0023050 con 0x7f63e005c7c0 2021-03-09T09:06:41.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:06:41.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:41.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b9f0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7f63e0016e10 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:41.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:06:41.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.062+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04f9d0 con 0x7f63e005c7c0 2021-03-09T09:06:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.063+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.063+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.063+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:41.078 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:41.078 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:41.078 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:41.078 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:41.079 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:41.079 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:41.079 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:41.079 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:41.079 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:41.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f63e005c7c0 2021-03-09T09:06:41.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.063+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:41.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.063+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:06:41.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f63e005c7c0 2021-03-09T09:06:41.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:41.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:41.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:42.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:42.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:42.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:42.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:42.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:42.476+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:43.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:43.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:43.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:43.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:44.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:44.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:44.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:44.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:44.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:44.550+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:45.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:45.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:46.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:46.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e000b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:46.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b9f0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7f63e0016e10 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:46.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b9f0 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 rx=0x7f63e0016e10 tx=0x7f63fc003a50).stop 2021-03-09T09:06:46.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.061+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:46.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.061+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:46.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.061+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e000b9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:46.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.061+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b9f0 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:46.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:06:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:46.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:46.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:46.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:46.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015a00 con 0x7f63e00070a0 2021-03-09T09:06:46.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:46.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ce40 con 0x7f6400145bd0 2021-03-09T09:06:46.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:46.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078f60 con 0x7f6400146050 2021-03-09T09:06:46.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:46.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:46.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:46.071 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7909 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:46.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:46.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:46.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:46.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:06:46.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:46.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.062+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:06:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:06:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000da10 con 0x7f6400146050 2021-03-09T09:06:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00786a0 con 0x7f6400146050 2021-03-09T09:06:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:06:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.063+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.065+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a2c0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40093c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.065+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:06:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.065+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022990 con 0x7f6400146050 2021-03-09T09:06:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.066+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.066+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:46.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.066+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:46.080 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:46.080 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:46.080 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:46.080 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:46.082 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f6400146050 2021-03-09T09:06:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.067+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022990 con 0x7f6400146050 2021-03-09T09:06:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.068+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:06:46.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:46.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:46.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:47.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:47.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:47.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:47.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:48.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:48.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:48.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:48.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:49.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:49.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:49.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:49.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:49.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:49.551+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:50.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:50.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:50.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:50.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:06:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a2c0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40093c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:51.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a2c0 secure :-1 s=READY pgs=2996 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40093c0).stop 2021-03-09T09:06:51.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:51.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:51.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002a2c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:51.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002a2c0 unknown :-1 s=CLOSED pgs=2996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:51.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:06:51.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:51.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:51.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0022250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:51.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:51.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:51.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:51.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:51.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b8b0 con 0x7f63e00070a0 2021-03-09T09:06:51.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:51.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:06:51.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:51.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f000 con 0x7f63e005c7c0 2021-03-09T09:06:51.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:51.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:51.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.063+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:51.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.064+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:51.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.064+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:51.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.064+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0022250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:51.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.064+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:06:51.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.064+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:51.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.064+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:51.073 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:51.064+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7936 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:51.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.065+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:51.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.065+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:51.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.065+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:51.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:51.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:51.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:51.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:51.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:51.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:06:51.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:06:51.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:51.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:51.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022740 con 0x7f63e005c7c0 2021-03-09T09:06:51.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:51.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:51.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000bee0 con 0x7f63e005c7c0 2021-03-09T09:06:51.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:06:51.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.066+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:51.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.067+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:06:51.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.067+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:51.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.067+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0022250 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc052f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:51.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e005c7c0 2021-03-09T09:06:51.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.068+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:51.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.068+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:51.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.068+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:51.081 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:51.081 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:51.081 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:51.083 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e005c7c0 2021-03-09T09:06:51.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.070+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:51.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.070+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:06:51.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f63e005c7c0 2021-03-09T09:06:51.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:51.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:51.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:52.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:52.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:52.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:52.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:52.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:52.477+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:53.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:53.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:53.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:53.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:54.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:54.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:54.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:54.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:54.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:54.552+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:06:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:55.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:55.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:56.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.064+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0022250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:06:56.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.064+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0022250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:06:56.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0022250 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc052f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:06:56.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0022250 secure :-1 s=READY pgs=3011 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc052f30).stop 2021-03-09T09:06:56.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:56.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:06:56.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0022250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:06:56.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0022250 unknown :-1 s=CLOSED pgs=3011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:06:56.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:06:56.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:56.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:06:56.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:56.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:06:56.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:06:56.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:06:56.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:56.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00292a0 con 0x7f63e00070a0 2021-03-09T09:06:56.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:56.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b3f0 con 0x7f6400145bd0 2021-03-09T09:06:56.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:06:56.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00658e0 con 0x7f6400146050 2021-03-09T09:06:56.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:06:56.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:06:56.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:56.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:06:56.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:06:56.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:06:56.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.065+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:06:56.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.066+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:06:56.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.066+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:06:56.075 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:06:56.066+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 7982 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:06:56.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.066+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:06:56.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.066+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:06:56.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.066+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:06:56.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:06:56.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:56.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:56.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:06:56.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:06:56.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:06:56.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:06:56.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:06:56.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:56.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ab60 con 0x7f6400146050 2021-03-09T09:06:56.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:06:56.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:06:56.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022740 con 0x7f6400146050 2021-03-09T09:06:56.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:06:56.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.068+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:56.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c1f0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc02d2f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:06:56.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.069+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:06:56.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.069+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:06:56.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0327c0 con 0x7f6400146050 2021-03-09T09:06:56.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.069+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:06:56.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.069+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:06:56.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.069+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:06:56.083 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:06:56.083 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:06:56.083 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:56.083 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:06:56.083 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:06:56.084 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:06:56.084 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:06:56.084 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:06:56.084 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:06:56.084 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:06:56.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0327c0 con 0x7f6400146050 2021-03-09T09:06:56.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.070+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:06:56.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.070+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:06:56.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:06:56.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:56.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:56.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:57.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:57.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:57.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:57.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:58.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:58.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:58.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:59.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:06:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:59.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:06:59.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:06:59.553+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:00.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:00.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:00.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:00.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:01.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:01.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:01.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c1f0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc02d2f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:01.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c1f0 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc02d2f0).stop 2021-03-09T09:07:01.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:01.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:01.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000c1f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:01.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000c1f0 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:01.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:07:01.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0018790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:01.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:01.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:01.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:01.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:01.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:01.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:01.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013a80 con 0x7f63e00070a0 2021-03-09T09:07:01.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:01.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078f60 con 0x7f6400145bd0 2021-03-09T09:07:01.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:01.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:07:01.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:01.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:01.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.066+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:01.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.067+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:01.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.067+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:01.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.067+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0018790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:01.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.067+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:07:01.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.067+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:01.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.067+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:01.076 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:01.067+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8014 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:01.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:01.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:01.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:01.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:01.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:01.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:01.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:01.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:01.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:07:01.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:07:01.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:01.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:01.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d1a0 con 0x7f63e005c7c0 2021-03-09T09:07:01.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:01.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:01.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000da10 con 0x7f63e005c7c0 2021-03-09T09:07:01.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:07:01.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.068+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:01.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.069+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0018790 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f4009910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:01.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.069+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:07:01.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.069+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:01.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f403fe50 con 0x7f63e005c7c0 2021-03-09T09:07:01.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.069+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:01.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.069+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:01.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.069+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:01.084 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:01.084 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:01.084 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:01.085 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:01.085 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:01.085 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:01.085 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:01.085 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:01.086 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:01.086 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:01.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e005c7c0 2021-03-09T09:07:01.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.069+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:01.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.070+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:07:01.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e005c7c0 2021-03-09T09:07:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:01.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:02.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:02.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:02.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:02.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:02.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:02.479+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:03.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:03.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:03.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:03.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:04.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:04.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:04.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:04.554+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:05.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:05.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:05.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:05.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:06.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0018790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:06.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0018790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:06.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0018790 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f4009910).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:06.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0018790 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f4009910).stop 2021-03-09T09:07:06.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:06.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:06.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0018790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:06.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0018790 unknown :-1 s=CLOSED pgs=3035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:06.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:07:06.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:06.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:06.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:06.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:06.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:06.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:06.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:06.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f000 con 0x7f63e00070a0 2021-03-09T09:07:06.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:06.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001910 con 0x7f6400145bd0 2021-03-09T09:07:06.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:06.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006cd20 con 0x7f6400146050 2021-03-09T09:07:06.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:06.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:06.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:06.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:06.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:06.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:06.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:07:06.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:06.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:06.078 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:06.068+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8060 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:06.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:06.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:06.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:06.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:06.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:06.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:06.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:06.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:06.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:07:06.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:07:06.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:06.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:06.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065730 con 0x7f6400146050 2021-03-09T09:07:06.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:06.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:06.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ab60 con 0x7f6400146050 2021-03-09T09:07:06.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:07:06.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:06.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007e3d0 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f63fc032520 tx=0x7f63fc047130).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:06.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:07:06.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.069+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:06.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00f9a0 con 0x7f6400146050 2021-03-09T09:07:06.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.071+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:06.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.071+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:06.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.071+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:06.085 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:06.085 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:06.085 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:06.086 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:06.086 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:06.086 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:06.086 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:06.087 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:06.087 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:06.087 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:06.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00fe60 con 0x7f6400146050 2021-03-09T09:07:06.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.071+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:06.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.071+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:07:06.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021250 con 0x7f6400146050 2021-03-09T09:07:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:06.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:06.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:07.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:07.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:07.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:07.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:08.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:08.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:08.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:08.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:09.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:09.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:09.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:09.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:09.555+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:10.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:10.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:10.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:10.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:11.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:07:11.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:11.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007e3d0 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f63fc032520 tx=0x7f63fc047130).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:11.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007e3d0 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 rx=0x7f63fc032520 tx=0x7f63fc047130).stop 2021-03-09T09:07:11.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:11.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:11.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007e3d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:11.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007e3d0 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:11.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:07:11.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:11.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:11.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00235d0 con 0x7f63e00070a0 2021-03-09T09:07:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e1c0 con 0x7f6400145bd0 2021-03-09T09:07:11.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001da00 con 0x7f63e005c7c0 2021-03-09T09:07:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:11.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:11.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.069+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:11.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:11.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:07:11.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:11.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:11.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:11.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:11.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:11.081 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8094 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:11.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:11.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:07:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:07:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:11.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:07:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.070+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d1a0 con 0x7f63e005c7c0 2021-03-09T09:07:11.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.071+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:07:11.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.071+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:11.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.071+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e8c0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 rx=0x7f63f4009c30 tx=0x7f63f40035e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:11.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.071+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:07:11.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.071+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:11.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.073+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40341b0 con 0x7f63e005c7c0 2021-03-09T09:07:11.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.073+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:11.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.073+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:11.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.073+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:11.088 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:11.088 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:11.088 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:11.088 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:11.088 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:11.089 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:11.089 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40341b0 con 0x7f63e005c7c0 2021-03-09T09:07:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.075+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.075+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:07:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:07:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:11.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:12.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:12.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:12.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:12.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:12.480+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:13.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:13.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:13.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:13.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:14.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:14.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:14.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:14.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:14.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:14.556+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:15.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:15.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:15.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:15.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:16.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:16.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:16.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e8c0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 rx=0x7f63f4009c30 tx=0x7f63f40035e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:16.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e8c0 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 rx=0x7f63f4009c30 tx=0x7f63f40035e0).stop 2021-03-09T09:07:16.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:16.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:16.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007e8c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:16.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e8c0 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:16.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:07:16.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:16.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:16.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:16.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:16.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:16.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:16.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:16.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:07:16.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:16.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b3f0 con 0x7f6400145bd0 2021-03-09T09:07:16.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:16.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018100 con 0x7f6400146050 2021-03-09T09:07:16.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.070+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:16.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.071+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:16.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.071+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:16.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.071+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:16.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.071+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:07:16.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.071+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:16.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.071+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:16.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.071+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:16.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.071+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:16.081 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:16.071+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8135 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:16.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:16.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:16.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:16.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:16.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:16.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:16.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:07:16.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:07:16.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:16.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:16.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ab60 con 0x7f6400146050 2021-03-09T09:07:16.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:16.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:16.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065730 con 0x7f6400146050 2021-03-09T09:07:16.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:07:16.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.072+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:16.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.073+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027a80 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7f63fc04fe90 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:16.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.073+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:07:16.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.073+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:16.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.077+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f6400146050 2021-03-09T09:07:16.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.077+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:16.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.077+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:16.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.077+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:16.091 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:16.091 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:16.091 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:16.091 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:16.092 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:16.092 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:16.092 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:16.093 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:16.093 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:16.093 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:16.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.077+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d370 con 0x7f6400146050 2021-03-09T09:07:16.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.077+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:16.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.077+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:07:16.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04fb90 con 0x7f6400146050 2021-03-09T09:07:16.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:16.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:16.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:17.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:17.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:17.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:17.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:18.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:18.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:18.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:18.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:19.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:19.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:19.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:19.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:19.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:19.557+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:20.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:20.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:20.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:20.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:21.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:21.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:21.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027a80 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7f63fc04fe90 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:21.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027a80 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 rx=0x7f63fc04fe90 tx=0x7f63fc009990).stop 2021-03-09T09:07:21.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:21.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:21.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:21.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027a80 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:21.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:07:21.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:21.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:21.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:21.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:21.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00127e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:21.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:21.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:21.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e00070a0 2021-03-09T09:07:21.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:21.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017710 con 0x7f6400145bd0 2021-03-09T09:07:21.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:21.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001b10 con 0x7f63e005c7c0 2021-03-09T09:07:21.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:21.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:21.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:21.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:21.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.072+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:21.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:21.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.073+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:07:21.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.073+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:21.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.073+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:21.084 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:21.073+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8177 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:21.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:21.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:21.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:21.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:21.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:21.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:21.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:21.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:21.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:07:21.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:07:21.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:21.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:21.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012cd0 con 0x7f63e005c7c0 2021-03-09T09:07:21.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:21.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:21.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012650 con 0x7f63e005c7c0 2021-03-09T09:07:21.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:07:21.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.074+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:21.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.075+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00127e0 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 rx=0x7f63f0055c80 tx=0x7f63f0054310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:21.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.075+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:07:21.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.075+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:21.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055cb0 con 0x7f63e005c7c0 2021-03-09T09:07:21.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.076+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:21.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.076+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:21.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.076+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:21.092 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:21.092 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:21.092 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:21.093 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:21.093 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:21.093 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:21.093 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:21.093 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:21.094 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:21.094 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:21.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.077+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001a1e0 con 0x7f63e005c7c0 2021-03-09T09:07:21.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.077+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.077+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:07:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e005c7c0 2021-03-09T09:07:21.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:21.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:21.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:22.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:22.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:22.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:22.481+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:23.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:23.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:23.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:23.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:24.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:24.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:24.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:24.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:24.558+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:25.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:25.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:25.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:25.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:26.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00127e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:26.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00127e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:26.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00127e0 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 rx=0x7f63f0055c80 tx=0x7f63f0054310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:26.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00127e0 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 rx=0x7f63f0055c80 tx=0x7f63f0054310).stop 2021-03-09T09:07:26.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:26.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:26.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00127e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:26.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00127e0 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:26.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:07:26.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:26.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:26.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:26.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:26.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:26.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:26.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:26.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e1c0 con 0x7f63e00070a0 2021-03-09T09:07:26.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:26.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e00 con 0x7f6400145bd0 2021-03-09T09:07:26.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:26.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:07:26.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:26.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:26.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.073+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:26.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:26.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:26.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:26.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:07:26.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:26.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:26.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:26.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:26.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:26.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:26.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:26.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:26.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:26.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:26.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:07:26.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:07:26.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:26.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:26.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019c30 con 0x7f6400146050 2021-03-09T09:07:26.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:26.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:26.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:07:26.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:07:26.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.075+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:26.088 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:26.074+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8204 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:26.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:07:26.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019740 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 rx=0x7f63f4022ca0 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:26.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:26.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401aaf0 con 0x7f6400146050 2021-03-09T09:07:26.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:26.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:26.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:26.092 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:26.092 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:26.092 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:26.092 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:26.093 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:26.093 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:26.093 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:26.094 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:26.094 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:26.094 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:26.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f401ac70 con 0x7f6400146050 2021-03-09T09:07:26.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:26.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.076+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:07:26.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:07:26.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:26.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:26.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:27.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:27.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:28.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:28.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:28.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:28.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:29.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:29.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:29.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:29.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:29.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:29.559+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:30.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:30.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:30.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:30.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:31.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:31.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:31.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019740 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 rx=0x7f63f4022ca0 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:31.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019740 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 rx=0x7f63f4022ca0 tx=0x7f63f4017ed0).stop 2021-03-09T09:07:31.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:31.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:31.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:31.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019740 unknown :-1 s=CLOSED pgs=3102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:31.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:07:31.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:31.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:31.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:31.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:31.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:31.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:31.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:31.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b650 con 0x7f63e00070a0 2021-03-09T09:07:31.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:31.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400145bd0 2021-03-09T09:07:31.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:31.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e005c7c0 2021-03-09T09:07:31.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:31.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:31.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.075+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:31.084 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8248 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:31.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:31.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:07:31.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:31.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:31.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:31.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:31.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:31.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:31.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:31.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:31.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:31.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:31.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:31.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:31.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:07:31.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:07:31.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:31.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:31.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cb90 con 0x7f63e005c7c0 2021-03-09T09:07:31.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:31.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:31.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012cd0 con 0x7f63e005c7c0 2021-03-09T09:07:31.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:07:31.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.076+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:31.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.077+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e710 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 rx=0x7f63f0065810 tx=0x7f63f001b330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:31.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.077+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:07:31.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.077+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:31.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001a2b0 con 0x7f63e005c7c0 2021-03-09T09:07:31.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.078+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:31.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.078+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:31.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.078+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:31.092 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:31.093 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:31.093 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:31.093 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:31.093 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:31.093 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:31.094 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:31.094 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:31.094 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:31.094 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:31.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0065240 con 0x7f63e005c7c0 2021-03-09T09:07:31.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.078+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:31.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.078+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:07:31.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0065240 con 0x7f63e005c7c0 2021-03-09T09:07:31.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:31.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:31.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:32.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:32.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:32.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:32.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:32.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:32.482+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:33.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:33.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:33.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:33.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:34.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:34.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:34.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:34.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:34.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:34.560+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:35.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:35.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:35.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:35.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:36.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:36.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:36.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e710 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 rx=0x7f63f0065810 tx=0x7f63f001b330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:36.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e710 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 rx=0x7f63f0065810 tx=0x7f63f001b330).stop 2021-03-09T09:07:36.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:36.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:36.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007e710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:36.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007e710 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:36.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:07:36.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:36.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:36.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:36.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:36.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:36.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:36.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017710 con 0x7f63e00070a0 2021-03-09T09:07:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ab00 con 0x7f6400145bd0 2021-03-09T09:07:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:36.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001b10 con 0x7f6400146050 2021-03-09T09:07:36.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:36.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:36.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.076+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:36.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.077+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:36.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.077+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:36.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.077+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:36.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.077+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:07:36.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.077+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:36.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.077+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:36.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:36.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:36.087 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:36.077+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8284 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:36.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:36.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:36.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:36.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:36.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:36.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:36.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:07:36.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:07:36.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:36.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:36.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:07:36.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:36.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:36.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019c30 con 0x7f6400146050 2021-03-09T09:07:36.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:07:36.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.078+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:36.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.079+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013040 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 rx=0x7f63f402a210 tx=0x7f63f404bf70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:36.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.079+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:07:36.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.079+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:36.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:07:36.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.080+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:36.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.080+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:36.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.080+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:36.093 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:36.094 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:36.094 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:36.094 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:36.094 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:36.094 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:36.095 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:36.095 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:36.095 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:36.095 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:36.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:07:36.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.081+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:36.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.081+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:07:36.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.082+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T09:07:36.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:36.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:36.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:37.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:37.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:37.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:37.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:38.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:38.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:38.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:38.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:39.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:39.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:39.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:39.561+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:40.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:40.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:40.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:41.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0013040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:41.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0013040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:41.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013040 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 rx=0x7f63f402a210 tx=0x7f63f404bf70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:41.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013040 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 rx=0x7f63f402a210 tx=0x7f63f404bf70).stop 2021-03-09T09:07:41.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:41.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:41.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0013040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:41.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0013040 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:41.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:07:41.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:41.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:41.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:41.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:41.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:41.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:41.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:41.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3f0 con 0x7f63e00070a0 2021-03-09T09:07:41.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:41.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e00 con 0x7f6400145bd0 2021-03-09T09:07:41.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:41.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b750 con 0x7f63e005c7c0 2021-03-09T09:07:41.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:41.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:41.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:41.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:41.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:07:41.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:41.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:41.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:41.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.078+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:41.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.079+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:41.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.079+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:41.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.079+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:41.089 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:41.079+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8325 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:41.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:41.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:41.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:41.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:41.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:41.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:07:41.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:07:41.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:41.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:41.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00278a0 con 0x7f63e005c7c0 2021-03-09T09:07:41.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:41.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:41.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cb90 con 0x7f63e005c7c0 2021-03-09T09:07:41.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:07:41.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:41.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:07:41.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:41.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.080+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b7f0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 rx=0x7f63f0000ca0 tx=0x7f63f0013f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:41.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0065450 con 0x7f63e005c7c0 2021-03-09T09:07:41.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.081+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:41.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.081+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:41.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.081+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:41.096 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:41.096 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:41.096 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:41.097 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:41.097 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:41.097 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:41.097 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:41.098 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:41.098 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:41.098 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:41.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00489b0 con 0x7f63e005c7c0 2021-03-09T09:07:41.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.081+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:41.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.081+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:07:41.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f63e005c7c0 2021-03-09T09:07:41.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:41.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:41.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:42.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:42.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:42.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:42.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:42.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:42.484+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:43.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:43.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:43.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:43.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:44.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:44.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:44.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:44.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:44.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:44.562+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:45.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:45.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:45.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:45.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e000b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:07:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e000b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b7f0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 rx=0x7f63f0000ca0 tx=0x7f63f0013f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b7f0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 rx=0x7f63f0000ca0 tx=0x7f63f0013f70).stop 2021-03-09T09:07:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e000b7f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e000b7f0 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:07:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00792d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:46.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:46.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:46.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016a70 con 0x7f63e00070a0 2021-03-09T09:07:46.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:46.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001b10 con 0x7f6400145bd0 2021-03-09T09:07:46.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:46.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ab00 con 0x7f6400146050 2021-03-09T09:07:46.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:46.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:46.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:46.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:46.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.080+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:46.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00792d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:46.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.081+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:07:46.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.081+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:46.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.081+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:46.090 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:46.081+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8358 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:46.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.081+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:46.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.081+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:46.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.081+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:46.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:46.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:46.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:46.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:46.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:46.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:07:46.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:07:46.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:46.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:46.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000da10 con 0x7f6400146050 2021-03-09T09:07:46.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:46.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:46.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00278a0 con 0x7f6400146050 2021-03-09T09:07:46.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:07:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.082+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.083+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00792d0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f005ff60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.083+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:07:46.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.083+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:46.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.085+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040a60 con 0x7f6400146050 2021-03-09T09:07:46.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.085+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:46.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.085+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:46.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.085+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:46.096 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:46.097 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:46.098 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:46.098 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:46.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005df00 con 0x7f6400146050 2021-03-09T09:07:46.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.089+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:46.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f6400146050 2021-03-09T09:07:46.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.089+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:07:46.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:46.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:46.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:47.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:47.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:47.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:47.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:48.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:48.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:49.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:49.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:49.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:49.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:49.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:49.563+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:50.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:50.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:50.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:50.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:51.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00792d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:07:51.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00792d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:51.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00792d0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f005ff60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:51.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00792d0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f005ff60).stop 2021-03-09T09:07:51.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.081+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:51.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:51.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00792d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:51.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00792d0 unknown :-1 s=CLOSED pgs=3163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:51.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:07:51.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:51.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:51.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024c70 con 0x7f63e00070a0 2021-03-09T09:07:51.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:51.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e410 con 0x7f6400145bd0 2021-03-09T09:07:51.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:51.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:07:51.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:51.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:51.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:51.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:07:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:51.092 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:51.082+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8392 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:51.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:51.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:51.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:51.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:51.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:51.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:51.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:51.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:51.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:07:51.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:07:51.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:51.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:51.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00095f0 con 0x7f63e005c7c0 2021-03-09T09:07:51.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:51.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:51.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ab60 con 0x7f63e005c7c0 2021-03-09T09:07:51.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:07:51.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.083+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:51.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.084+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005e540 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 rx=0x7f63e0015d70 tx=0x7f63fc006d40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:51.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.084+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:07:51.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.084+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:51.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.085+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065e90 con 0x7f63e005c7c0 2021-03-09T09:07:51.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.085+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:51.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.085+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:51.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.085+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:51.099 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:51.099 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:51.099 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:51.100 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:51.100 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:51.100 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:51.100 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:51.101 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:51.101 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:51.101 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:51.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065e90 con 0x7f63e005c7c0 2021-03-09T09:07:51.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.087+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:51.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.087+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:07:51.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:07:51.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:51.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:51.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:52.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:52.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:52.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:52.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:52.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:52.485+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:53.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:53.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:53.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:53.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:54.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:54.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:54.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:54.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:54.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:54.564+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:07:55.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:55.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:55.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:55.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:56.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e005e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:07:56.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e005e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:07:56.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005e540 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 rx=0x7f63e0015d70 tx=0x7f63fc006d40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:07:56.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005e540 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 rx=0x7f63e0015d70 tx=0x7f63fc006d40).stop 2021-03-09T09:07:56.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:56.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:07:56.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e005e540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:07:56.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e005e540 unknown :-1 s=CLOSED pgs=3170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:07:56.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:07:56.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:56.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:07:56.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:56.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:07:56.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:07:56.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:07:56.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:56.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e00 con 0x7f63e00070a0 2021-03-09T09:07:56.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:56.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f000 con 0x7f6400145bd0 2021-03-09T09:07:56.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:07:56.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:07:56.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:07:56.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:07:56.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:56.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:07:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.083+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:07:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.084+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:07:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.084+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:07:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.084+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:07:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.084+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:07:56.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.084+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:07:56.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.084+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:07:56.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.084+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:07:56.093 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:07:56.084+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8431 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:07:56.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:07:56.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:07:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:07:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:07:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:07:56.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:07:56.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:56.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:07:56.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:07:56.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:07:56.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:07:56.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:07:56.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bb80 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f4003c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:07:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:07:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.085+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:07:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4046e80 con 0x7f6400146050 2021-03-09T09:07:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.086+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:07:56.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.086+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:07:56.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.086+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:07:56.100 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:07:56.100 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:07:56.101 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:56.101 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:07:56.101 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:07:56.101 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:07:56.101 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:07:56.102 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:07:56.102 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:07:56.102 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:07:56.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046e80 con 0x7f6400146050 2021-03-09T09:07:56.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.086+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:07:56.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.086+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:07:56.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f6400146050 2021-03-09T09:07:56.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:56.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:56.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:57.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:57.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:57.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:57.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:58.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:58.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:58.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:58.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:59.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:59.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:07:59.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:59.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:07:59.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:07:59.565+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:00.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:00.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:00.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:00.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:01.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:01.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:01.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bb80 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f4003c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:01.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bb80 secure :-1 s=READY pgs=3184 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f4003c20).stop 2021-03-09T09:08:01.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:01.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:01.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006bb80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:01.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006bb80 unknown :-1 s=CLOSED pgs=3184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:01.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:08:01.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0079b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:01.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:01.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:01.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:01.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:01.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:01.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:01.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b750 con 0x7f63e00070a0 2021-03-09T09:08:01.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:01.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:08:01.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:01.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e410 con 0x7f63e005c7c0 2021-03-09T09:08:01.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.084+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:01.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.085+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:01.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.085+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:01.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.085+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:01.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.085+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:01.093 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:01.085+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8467 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:01.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.086+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0079b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:01.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.086+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:08:01.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.086+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:01.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.086+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:01.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.086+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:01.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.086+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:01.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.086+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:01.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:01.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:01.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:01.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:01.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:01.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:08:01.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:08:01.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bfb0 con 0x7f63e005c7c0 2021-03-09T09:08:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f820 con 0x7f63e005c7c0 2021-03-09T09:08:01.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:08:01.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.088+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:01.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.089+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:08:01.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.089+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:01.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.089+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0079b20 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40051f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40624d0 con 0x7f63e005c7c0 2021-03-09T09:08:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.093+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.093+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.093+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:01.102 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:01.102 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:01.102 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:01.102 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:01.103 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:01.103 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:01.103 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:01.103 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:01.103 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:01.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4062650 con 0x7f63e005c7c0 2021-03-09T09:08:01.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.093+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:01.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.093+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:08:01.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:08:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:01.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:01.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:02.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:02.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:02.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:02.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:02.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:02.486+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:03.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:03.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:03.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:04.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:04.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:04.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:04.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:04.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:04.566+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:05.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:05.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:05.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:05.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:06.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0079b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:08:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0079b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0079b20 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40051f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0079b20 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40051f0).stop 2021-03-09T09:08:06.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:06.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:06.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0079b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:06.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0079b20 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:06.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:08:06.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:06.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00278a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b5f0 con 0x7f63e00070a0 2021-03-09T09:08:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:06.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00248f0 con 0x7f6400145bd0 2021-03-09T09:08:06.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:06.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e00 con 0x7f6400146050 2021-03-09T09:08:06.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.086+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:06.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.087+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:06.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.087+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:06.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.087+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:06.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.087+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:06.095 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:06.087+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8507 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:06.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.087+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00278a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:06.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.087+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:08:06.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.087+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:06.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.087+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:06.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:06.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:08:06.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:08:06.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:06.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:06.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025b50 con 0x7f6400146050 2021-03-09T09:08:06.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:06.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:06.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000da10 con 0x7f6400146050 2021-03-09T09:08:06.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:08:06.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:06.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00278a0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004cf50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:06.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:08:06.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.089+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:06.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.090+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040ec0 con 0x7f6400146050 2021-03-09T09:08:06.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.090+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:06.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.090+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:06.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.090+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:06.104 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:06.106 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040ec0 con 0x7f6400146050 2021-03-09T09:08:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.093+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.093+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:08:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f6400146050 2021-03-09T09:08:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:06.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:07.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:07.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:07.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:07.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:08.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:08.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:08.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:08.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:09.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:09.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:09.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:09.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:09.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:09.567+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:10.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:10.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:10.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:10.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:11.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.087+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00278a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:11.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.087+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00278a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:11.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.087+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00278a0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004cf50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:11.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.087+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00278a0 secure :-1 s=READY pgs=3216 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004cf50).stop 2021-03-09T09:08:11.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:11.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:11.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00278a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:11.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00278a0 unknown :-1 s=CLOSED pgs=3216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:11.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:08:11.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:11.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:11.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:11.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:11.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:11.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:11.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:11.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63e00070a0 2021-03-09T09:08:11.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:11.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e280 con 0x7f6400145bd0 2021-03-09T09:08:11.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:11.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029c50 con 0x7f63e005c7c0 2021-03-09T09:08:11.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:11.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:11.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:08:11.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:11.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.088+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:11.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:11.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:11.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:11.099 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8547 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:11.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:11.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:11.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:11.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:11.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:11.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:08:11.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:08:11.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:11.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:11.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0078db0 con 0x7f63e005c7c0 2021-03-09T09:08:11.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:11.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:11.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025b50 con 0x7f63e005c7c0 2021-03-09T09:08:11.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.090+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:08:11.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.090+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:11.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.090+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 rx=0x7f63e0019650 tx=0x7f63f00212f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:11.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.090+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:08:11.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.090+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:11.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.091+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0054030 con 0x7f63e005c7c0 2021-03-09T09:08:11.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.091+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:11.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.091+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:11.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.091+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:11.107 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:11.107 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:11.107 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:11.108 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:11.108 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:11.108 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:11.108 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:11.108 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:11.109 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:11.109 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:11.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.092+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0021750 con 0x7f63e005c7c0 2021-03-09T09:08:11.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.092+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:11.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.092+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:08:11.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.092+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0054030 con 0x7f63e005c7c0 2021-03-09T09:08:11.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:11.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:11.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:12.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:12.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:12.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:12.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:12.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:12.488+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:13.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:13.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:13.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:13.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:14.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:14.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:14.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:14.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:14.568+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:15.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:15.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:15.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:15.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:16.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0015c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:16.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0015c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:16.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 rx=0x7f63e0019650 tx=0x7f63f00212f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:16.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 secure :-1 s=READY pgs=3228 cs=0 l=1 rev1=1 rx=0x7f63e0019650 tx=0x7f63f00212f0).stop 2021-03-09T09:08:16.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:16.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:16.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0015c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:16.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 unknown :-1 s=CLOSED pgs=3228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:16.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:08:16.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:16.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:16.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:16.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:16.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:16.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:16.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:16.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027710 con 0x7f63e00070a0 2021-03-09T09:08:16.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:16.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e410 con 0x7f6400145bd0 2021-03-09T09:08:16.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:16.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:08:16.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:16.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:16.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:16.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:16.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.089+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:16.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.090+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:16.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.090+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:08:16.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.090+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:16.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.090+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:16.099 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:16.090+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8574 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:16.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.090+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:16.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.090+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:16.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.090+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:16.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.090+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:16.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:16.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:16.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:16.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:16.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:08:16.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:08:16.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:16.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:16.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026670 con 0x7f6400146050 2021-03-09T09:08:16.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:16.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:16.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bfb0 con 0x7f6400146050 2021-03-09T09:08:16.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:08:16.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.091+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:16.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.092+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:08:16.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.092+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 rx=0x7f63f4003620 tx=0x7f63f4070170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:16.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.092+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:16.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f6400146050 2021-03-09T09:08:16.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.097+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:16.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.097+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:16.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.097+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:16.106 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:16.106 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:16.106 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:16.107 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:16.107 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:16.107 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:16.107 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:16.107 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:16.108 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:16.108 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:16.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f6400146050 2021-03-09T09:08:16.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.097+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:16.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.097+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:08:16.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:08:16.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:16.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:16.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:17.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:17.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:17.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:17.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:18.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:18.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:18.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:18.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:19.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:19.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:19.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:19.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:19.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:19.569+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:20.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:20.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:20.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:20.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:21.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:21.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 rx=0x7f63f4003620 tx=0x7f63f4070170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 rx=0x7f63f4003620 tx=0x7f63f4070170).stop 2021-03-09T09:08:21.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:21.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 unknown :-1 s=CLOSED pgs=3240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:08:21.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:21.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:21.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:21.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:21.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:21.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00237d0 con 0x7f63e00070a0 2021-03-09T09:08:21.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:21.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d040 con 0x7f6400145bd0 2021-03-09T09:08:21.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:21.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e280 con 0x7f63e005c7c0 2021-03-09T09:08:21.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:21.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:21.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:21.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.091+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:21.105 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:21.092+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8621 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:21.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.092+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:21.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.092+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:08:21.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.092+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:21.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.092+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:21.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.093+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:21.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.093+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:21.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.093+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:21.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:21.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:21.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:21.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:21.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:21.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:08:21.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:08:21.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:21.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:21.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ab60 con 0x7f63e005c7c0 2021-03-09T09:08:21.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:21.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:21.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0078db0 con 0x7f63e005c7c0 2021-03-09T09:08:21.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:08:21.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:21.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:08:21.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 rx=0x7f63f000a010 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:21.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.094+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:21.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e005c7c0 2021-03-09T09:08:21.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.095+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:21.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.095+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:21.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.095+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:21.118 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:21.118 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:21.118 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:21.118 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:21.120 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:21.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e005c7c0 2021-03-09T09:08:21.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.095+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:21.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.095+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:08:21.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.096+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004d640 con 0x7f63e005c7c0 2021-03-09T09:08:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:21.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:22.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:22.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:22.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:22.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:22.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:22.488+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:23.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:23.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:23.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:23.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:24.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:24.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:24.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:24.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:24.570+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:25.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:25.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:25.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:25.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:26.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0015c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:26.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0015c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:26.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 rx=0x7f63f000a010 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:26.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 rx=0x7f63f000a010 tx=0x7f63f0000c30).stop 2021-03-09T09:08:26.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:26.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:26.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0015c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:26.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c50 unknown :-1 s=CLOSED pgs=3258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:26.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:08:26.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:26.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:26.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:26.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:26.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:26.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:26.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:26.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029c50 con 0x7f63e00070a0 2021-03-09T09:08:26.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:26.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400145bd0 2021-03-09T09:08:26.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:26.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027710 con 0x7f6400146050 2021-03-09T09:08:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:26.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.093+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:26.102 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:26.094+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8658 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:26.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.094+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:08:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.094+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.094+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:26.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.095+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:26.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.095+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:26.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.095+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:26.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:26.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:26.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:26.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:26.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:26.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:08:26.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:08:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f9a0 con 0x7f6400146050 2021-03-09T09:08:26.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ab60 con 0x7f6400146050 2021-03-09T09:08:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:08:26.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.096+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.097+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010540 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f0030f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.097+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:08:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.097+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027760 con 0x7f6400146050 2021-03-09T09:08:26.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.098+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:26.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.098+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:26.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.098+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:26.110 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:26.110 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:26.110 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:26.111 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:26.111 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:26.111 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:26.111 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:26.112 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:26.112 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:26.112 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:26.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001ae30 con 0x7f6400146050 2021-03-09T09:08:26.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.100+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:26.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.100+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:08:26.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027760 con 0x7f6400146050 2021-03-09T09:08:26.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:26.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:26.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:27.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:27.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:28.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:28.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:28.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:28.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:29.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:29.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:29.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:29.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:29.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:29.571+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:30.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:30.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:30.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:30.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:31.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0010540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:31.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0010540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:31.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010540 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f0030f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:31.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010540 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f0030f20).stop 2021-03-09T09:08:31.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:31.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:31.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0010540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:31.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010540 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:31.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:08:31.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:31.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:31.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:31.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:31.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00278a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:31.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:31.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:31.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017710 con 0x7f63e00070a0 2021-03-09T09:08:31.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:31.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:08:31.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:31.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e410 con 0x7f63e005c7c0 2021-03-09T09:08:31.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:31.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:31.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:31.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:31.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:31.104 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:31.095+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8701 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:31.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.096+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00278a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:31.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.096+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:08:31.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.096+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:31.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.096+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:31.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:31.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:31.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:31.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:31.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:31.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:31.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:31.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:31.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:08:31.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:08:31.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:31.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:31.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001c4e0 con 0x7f63e005c7c0 2021-03-09T09:08:31.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:31.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:31.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f9a0 con 0x7f63e005c7c0 2021-03-09T09:08:31.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:08:31.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.097+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:31.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.098+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00278a0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f63f00090c0 tx=0x7f63f003feb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:31.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.098+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:08:31.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.098+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:31.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0021110 con 0x7f63e005c7c0 2021-03-09T09:08:31.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.102+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:31.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.102+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:31.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.102+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:31.112 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:31.112 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:31.112 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:31.112 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:31.113 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:31.113 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:31.113 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:31.113 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:31.113 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:31.114 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:31.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0021110 con 0x7f63e005c7c0 2021-03-09T09:08:31.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.103+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:31.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.103+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:08:31.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0054030 con 0x7f63e005c7c0 2021-03-09T09:08:31.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:31.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:31.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:32.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:32.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:32.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:32.490+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:33.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:33.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:33.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:33.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:34.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:34.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:34.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:34.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:34.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:34.572+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:35.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:35.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:35.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:35.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:36.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00278a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:36.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00278a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:36.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00278a0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f63f00090c0 tx=0x7f63f003feb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:36.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00278a0 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 rx=0x7f63f00090c0 tx=0x7f63f003feb0).stop 2021-03-09T09:08:36.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:36.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:36.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00278a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:36.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00278a0 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:36.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:08:36.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:36.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:36.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:36.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:36.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:36.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:36.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:36.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d040 con 0x7f63e00070a0 2021-03-09T09:08:36.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:36.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017710 con 0x7f6400145bd0 2021-03-09T09:08:36.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:36.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027710 con 0x7f6400146050 2021-03-09T09:08:36.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:36.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:36.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:36.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:36.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:08:36.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:36.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.096+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:36.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:36.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:36.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:36.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:36.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:36.107 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8730 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:36.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:36.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:36.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:36.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:36.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:36.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:08:36.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:08:36.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:36.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:36.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:08:36.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:36.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:36.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00095f0 con 0x7f6400146050 2021-03-09T09:08:36.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:08:36.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.097+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:36.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.098+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:08:36.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.098+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:36.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.098+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc046ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:36.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.099+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc076360 con 0x7f6400146050 2021-03-09T09:08:36.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.099+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:36.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.099+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:36.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.099+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:36.113 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:36.113 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:36.113 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:36.114 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:36.114 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:36.114 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:36.114 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:36.114 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:36.115 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:36.115 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:36.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.099+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc076130 con 0x7f6400146050 2021-03-09T09:08:36.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.099+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:36.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.099+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:08:36.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:08:36.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:36.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:36.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:37.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:37.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:37.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:37.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:38.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:38.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:38.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:38.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:39.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:39.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:39.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:39.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:39.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:39.573+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:40.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:40.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:40.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:40.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:41.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:41.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:41.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc046ae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:41.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 secure :-1 s=READY pgs=3297 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc046ae0).stop 2021-03-09T09:08:41.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:41.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:41.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:41.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023ac0 unknown :-1 s=CLOSED pgs=3297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:41.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:08:41.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:41.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:41.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:41.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:41.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:41.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:41.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:41.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e410 con 0x7f63e00070a0 2021-03-09T09:08:41.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:41.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:08:41.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:41.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077a40 con 0x7f63e005c7c0 2021-03-09T09:08:41.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:41.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:41.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:41.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:41.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.098+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:41.107 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:41.099+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8774 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:41.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.099+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:41.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.099+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:08:41.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.099+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:41.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.099+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:41.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.099+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:41.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.099+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:41.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.099+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:41.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:41.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:41.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:41.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:41.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:41.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:08:41.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:08:41.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:41.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:41.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ab60 con 0x7f63e005c7c0 2021-03-09T09:08:41.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:41.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:41.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001c4e0 con 0x7f63e005c7c0 2021-03-09T09:08:41.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:08:41.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.100+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:41.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.101+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:08:41.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.101+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:41.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.101+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 rx=0x7f63e006bae0 tx=0x7f63f0006a40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:41.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003fd40 con 0x7f63e005c7c0 2021-03-09T09:08:41.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.102+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:41.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.102+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:41.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.102+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:41.116 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:41.116 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:41.116 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:41.116 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:41.117 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:41.117 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:41.117 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:41.117 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:41.118 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:41.118 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:41.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003fd40 con 0x7f63e005c7c0 2021-03-09T09:08:41.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.104+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:41.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.104+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:08:41.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0049770 con 0x7f63e005c7c0 2021-03-09T09:08:41.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:41.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:41.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:42.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:42.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:42.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:42.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:42.492+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:43.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:43.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:43.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:43.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:44.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:44.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:44.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:44.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:44.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:44.574+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:45.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:45.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:45.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:45.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:46.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:08:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 rx=0x7f63e006bae0 tx=0x7f63f0006a40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:46.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 rx=0x7f63e006bae0 tx=0x7f63f0006a40).stop 2021-03-09T09:08:46.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:46.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:46.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002e3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:46.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 unknown :-1 s=CLOSED pgs=3317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:46.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:08:46.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:46.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:46.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:46.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:46.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:46.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000be10 con 0x7f63e00070a0 2021-03-09T09:08:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:46.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029e50 con 0x7f6400145bd0 2021-03-09T09:08:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018320 con 0x7f6400146050 2021-03-09T09:08:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:46.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:46.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:46.108 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8816 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:46.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:46.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:08:46.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:46.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:46.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.100+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:46.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:46.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:08:46.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:08:46.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:46.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:46.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f9a0 con 0x7f6400146050 2021-03-09T09:08:46.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:46.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:46.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026670 con 0x7f6400146050 2021-03-09T09:08:46.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:08:46.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.101+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:46.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.102+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e8b0 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4005190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:46.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.102+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:08:46.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.102+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:46.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f6400146050 2021-03-09T09:08:46.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.104+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:46.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.104+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:46.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.104+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:46.116 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:46.116 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:46.116 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:46.117 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:46.118 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:46.118 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:46.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005b60 con 0x7f6400146050 2021-03-09T09:08:46.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.104+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:46.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.104+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:08:46.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f6400146050 2021-03-09T09:08:46.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:46.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:46.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:47.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:47.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:47.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:47.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:48.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:48.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:49.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:49.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:49.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:49.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:49.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:49.575+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:50.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:50.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:50.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:50.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:51.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:51.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:51.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e8b0 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4005190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:51.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e8b0 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4005190).stop 2021-03-09T09:08:51.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:51.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:51.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e8b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:51.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e8b0 unknown :-1 s=CLOSED pgs=3326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:51.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:08:51.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:51.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:51.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:51.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:51.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:51.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:51.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:51.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63e00070a0 2021-03-09T09:08:51.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:51.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00090d0 con 0x7f6400145bd0 2021-03-09T09:08:51.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:51.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b140 con 0x7f63e005c7c0 2021-03-09T09:08:51.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:51.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:51.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:51.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:51.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:51.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:51.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:08:51.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:51.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.101+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:51.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:51.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:51.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:51.112 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8846 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:51.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:51.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:51.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:51.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:51.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:51.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:08:51.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:08:51.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:51.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:51.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a8d0 con 0x7f63e005c7c0 2021-03-09T09:08:51.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:51.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:51.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.102+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f9a0 con 0x7f63e005c7c0 2021-03-09T09:08:51.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.103+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:08:51.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.103+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:51.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.103+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023ac0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4047b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:51.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.103+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:08:51.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.103+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:51.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405bb20 con 0x7f63e005c7c0 2021-03-09T09:08:51.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.105+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:51.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.105+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:51.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.105+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:51.118 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:51.118 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:51.118 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:51.118 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:51.118 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:51.119 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:51.119 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:51.119 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:51.119 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:51.119 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:51.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009210 con 0x7f63e005c7c0 2021-03-09T09:08:51.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.106+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:51.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.106+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:08:51.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:08:51.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:51.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:51.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:52.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:52.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:52.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:52.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:52.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:52.492+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:53.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:53.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:53.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:53.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:54.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:54.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:54.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:54.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:54.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:54.576+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:08:55.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:55.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:55.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:55.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:56.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:08:56.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:08:56.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023ac0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4047b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:08:56.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023ac0 secure :-1 s=READY pgs=3338 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4047b00).stop 2021-03-09T09:08:56.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:56.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:08:56.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:08:56.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023ac0 unknown :-1 s=CLOSED pgs=3338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:08:56.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:08:56.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:56.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:08:56.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:56.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:08:56.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:08:56.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:08:56.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:56.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e410 con 0x7f63e00070a0 2021-03-09T09:08:56.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:56.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:08:56.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:08:56.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018320 con 0x7f6400146050 2021-03-09T09:08:56.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:08:56.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:08:56.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:08:56.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:56.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:08:56.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:08:56.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:08:56.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:08:56.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.103+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:08:56.113 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:08:56.104+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8883 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:08:56.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.104+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:08:56.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.104+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:08:56.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.104+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:08:56.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:08:56.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:56.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:56.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:08:56.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:08:56.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:08:56.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:08:56.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:08:56.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:56.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007a080 con 0x7f6400146050 2021-03-09T09:08:56.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:08:56.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:08:56.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:08:56.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:08:56.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:56.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e020 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 rx=0x7f63fc0471e0 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:08:56.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:08:56.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.105+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:08:56.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.107+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:08:56.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.107+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:08:56.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.107+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:08:56.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.107+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:08:56.120 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:08:56.120 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:08:56.121 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:56.121 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:08:56.121 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:08:56.121 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:08:56.121 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:08:56.122 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:08:56.122 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:08:56.122 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:08:56.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.107+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0711d0 con 0x7f6400146050 2021-03-09T09:08:56.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.107+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:08:56.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.107+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:08:56.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:08:56.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:56.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:56.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:57.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:57.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:57.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:58.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:58.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:58.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:58.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:59.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:59.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:08:59.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:59.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:08:59.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:08:59.577+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:00.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:00.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:00.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:00.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:01.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:09:01.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:01.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e020 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 rx=0x7f63fc0471e0 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:01.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e020 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 rx=0x7f63fc0471e0 tx=0x7f63fc04a740).stop 2021-03-09T09:09:01.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:01.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:01.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001e020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:01.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e020 unknown :-1 s=CLOSED pgs=3354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:01.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:09:01.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:01.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:01.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:01.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:01.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:01.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:01.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:01.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00090d0 con 0x7f63e00070a0 2021-03-09T09:09:01.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:01.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f6400145bd0 2021-03-09T09:09:01.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:01.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077840 con 0x7f63e005c7c0 2021-03-09T09:09:01.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:01.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:01.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:01.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.105+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:01.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.106+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:01.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.106+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:01.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.106+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:09:01.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.106+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:01.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.106+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:01.115 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:01.106+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8923 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:01.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.106+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:01.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.106+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:01.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.106+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:01.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:01.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:01.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:01.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:01.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:01.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:09:01.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:09:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f9a0 con 0x7f63e005c7c0 2021-03-09T09:09:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:01.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:01.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007a080 con 0x7f63e005c7c0 2021-03-09T09:09:01.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:09:01.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.107+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:01.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.108+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc02c290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:01.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.108+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:09:01.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.108+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:01.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.109+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00f140 con 0x7f63e005c7c0 2021-03-09T09:09:01.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.109+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:01.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.109+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:01.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.109+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:01.122 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:01.122 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:01.122 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:01.122 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:01.123 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:01.123 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:01.123 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:01.123 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:01.123 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:01.124 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:01.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00f140 con 0x7f63e005c7c0 2021-03-09T09:09:01.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.110+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:01.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.110+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:09:01.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e005c7c0 2021-03-09T09:09:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:01.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:02.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:02.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:02.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:02.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:02.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:02.495+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:03.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:03.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:03.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:03.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:04.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:04.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:04.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:04.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:04.578+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:05.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:05.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:05.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:05.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:06.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:06.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc02c290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc02c290).stop 2021-03-09T09:09:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002e3c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002e3c0 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:09:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cda0 con 0x7f63e00070a0 2021-03-09T09:09:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400145bd0 2021-03-09T09:09:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027710 con 0x7f6400146050 2021-03-09T09:09:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.107+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:06.115 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:06.108+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 8972 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.108+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.108+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:09:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.108+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.108+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.108+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.108+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.108+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:09:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:09:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018120 con 0x7f6400146050 2021-03-09T09:09:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f9a0 con 0x7f6400146050 2021-03-09T09:09:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:09:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc077670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:09:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.109+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:06.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0641a0 con 0x7f6400146050 2021-03-09T09:09:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.110+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.110+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.110+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:06.123 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:06.124 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:06.124 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:06.124 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:06.124 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:06.125 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:06.126 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0777c0 con 0x7f6400146050 2021-03-09T09:09:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.110+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:06.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.110+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:09:06.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f6400146050 2021-03-09T09:09:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:06.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:06.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:07.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:07.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:07.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:07.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:08.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:08.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:08.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:08.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:09.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:09.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:09.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:09.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:09.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:09.579+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:10.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:10.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:11.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:11.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:11.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc077670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:11.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc077670).stop 2021-03-09T09:09:11.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:11.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:11.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:11.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=3380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:11.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:09:11.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:11.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:11.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:11.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:09:11.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:11.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028320 con 0x7f6400145bd0 2021-03-09T09:09:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c500 con 0x7f63e005c7c0 2021-03-09T09:09:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.108+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.109+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.109+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:11.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.109+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:11.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.109+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:11.116 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:11.109+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9003 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:11.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.109+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:11.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.109+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:09:11.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.109+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:11.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.109+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:11.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.110+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:11.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.110+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:11.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.110+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:11.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:11.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:11.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:09:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:09:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:11.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:11.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00200b0 con 0x7f63e005c7c0 2021-03-09T09:09:11.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018120 con 0x7f63e005c7c0 2021-03-09T09:09:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:09:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c190 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f63fc0471e0 tx=0x7f63fc00f1b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.112+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:09:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.112+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc076e70 con 0x7f63e005c7c0 2021-03-09T09:09:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.112+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:11.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.112+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:11.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.112+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:11.125 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:11.125 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:11.127 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:11.127 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:11.127 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:11.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0641a0 con 0x7f63e005c7c0 2021-03-09T09:09:11.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.112+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:11.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.112+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:09:11.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f63e005c7c0 2021-03-09T09:09:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:11.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:12.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:12.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:12.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:12.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:12.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:12.496+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:13.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:13.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:13.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:13.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:14.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:14.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:14.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:14.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:14.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:14.580+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:15.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:15.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:15.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:15.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:16.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:16.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:16.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c190 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f63fc0471e0 tx=0x7f63fc00f1b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:16.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c190 secure :-1 s=READY pgs=3394 cs=0 l=1 rev1=1 rx=0x7f63fc0471e0 tx=0x7f63fc00f1b0).stop 2021-03-09T09:09:16.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:16.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:16.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:16.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c190 unknown :-1 s=CLOSED pgs=3394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:09:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:16.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:16.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:16.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:16.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:16.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025b20 con 0x7f63e00070a0 2021-03-09T09:09:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00090d0 con 0x7f6400145bd0 2021-03-09T09:09:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f6400146050 2021-03-09T09:09:16.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:16.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:16.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:16.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:16.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:09:16.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:16.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:16.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:16.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:16.121 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9041 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:16.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:09:16.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:09:16.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:16.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:16.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:09:16.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:16.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:16.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00200b0 con 0x7f6400146050 2021-03-09T09:09:16.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.112+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:09:16.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.112+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:16.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.112+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:09:16.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.112+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc01f5c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:16.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.112+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:16.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc076be0 con 0x7f6400146050 2021-03-09T09:09:16.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.117+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:16.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.117+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:16.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.117+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:16.128 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:16.128 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:16.128 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:16.128 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:16.129 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:16.129 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:16.129 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:16.129 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:16.130 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:16.130 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:16.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06c940 con 0x7f6400146050 2021-03-09T09:09:16.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.117+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:16.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.117+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:09:16.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f6400146050 2021-03-09T09:09:16.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:16.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:16.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:17.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:17.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:18.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:18.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:18.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:18.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:19.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:19.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:19.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:19.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:19.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:19.581+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:20.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:20.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:20.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:20.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:21.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:09:21.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:21.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc01f5c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:21.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 secure :-1 s=READY pgs=3407 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc01f5c0).stop 2021-03-09T09:09:21.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:21.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:21.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:21.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 unknown :-1 s=CLOSED pgs=3407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:21.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:09:21.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:21.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:21.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:21.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:21.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:21.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:21.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:21.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ded0 con 0x7f63e00070a0 2021-03-09T09:09:21.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:21.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00192e0 con 0x7f6400145bd0 2021-03-09T09:09:21.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:21.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:09:21.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:21.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:21.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:09:21.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:21.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.112+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:21.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:21.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:21.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:21.122 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9079 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:21.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:21.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:21.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:21.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:21.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:21.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:09:21.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:09:21.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:21.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:21.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026670 con 0x7f63e005c7c0 2021-03-09T09:09:21.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:21.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:21.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ab60 con 0x7f63e005c7c0 2021-03-09T09:09:21.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:09:21.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.113+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:21.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.114+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:09:21.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.114+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:21.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.114+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001ee30 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 rx=0x7f63e00580e0 tx=0x7f63f0013f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:21.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0048e40 con 0x7f63e005c7c0 2021-03-09T09:09:21.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.116+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:21.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.116+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:21.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.116+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:21.129 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:21.129 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:21.129 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:21.129 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:21.130 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:21.130 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:21.130 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:21.130 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:21.130 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:21.131 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:21.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004ce60 con 0x7f63e005c7c0 2021-03-09T09:09:21.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.116+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:21.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.117+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:09:21.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f63e005c7c0 2021-03-09T09:09:21.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:21.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:21.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:22.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:22.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:22.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:22.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:22.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:22.497+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:23.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:23.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:23.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:23.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:24.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:24.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:24.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:24.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:24.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:24.582+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:25.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:25.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:25.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:25.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:26.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:09:26.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:26.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001ee30 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 rx=0x7f63e00580e0 tx=0x7f63f0013f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:26.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001ee30 secure :-1 s=READY pgs=3421 cs=0 l=1 rev1=1 rx=0x7f63e00580e0 tx=0x7f63f0013f00).stop 2021-03-09T09:09:26.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:26.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:26.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001ee30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:26.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001ee30 unknown :-1 s=CLOSED pgs=3421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:26.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:09:26.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:26.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:26.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c500 con 0x7f63e00070a0 2021-03-09T09:09:26.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:26.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ee60 con 0x7f6400145bd0 2021-03-09T09:09:26.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:26.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:09:26.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.112+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:26.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.113+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:26.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.113+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:26.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.113+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:26.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.113+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:26.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.113+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:26.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.113+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:09:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.113+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:26.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.113+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:26.123 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9118 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:26.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:26.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:26.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:26.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:26.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:26.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:26.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:26.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:26.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:09:26.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:09:26.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.114+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:26.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.115+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:26.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.115+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00282d0 con 0x7f6400146050 2021-03-09T09:09:26.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.115+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:26.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.115+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:26.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.115+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026670 con 0x7f6400146050 2021-03-09T09:09:26.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.115+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:09:26.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.115+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:26.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.116+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:09:26.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.116+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:26.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.116+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 rx=0x7f63e006ce60 tx=0x7f63f0027ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:26.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0065a70 con 0x7f6400146050 2021-03-09T09:09:26.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.118+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:26.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.118+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:26.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.118+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:26.131 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:26.132 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:26.132 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:26.132 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:26.132 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:26.133 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:26.133 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:26.133 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:26.133 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:26.134 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:26.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:09:26.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.121+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:26.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.121+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:09:26.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.122+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0003b10 con 0x7f6400146050 2021-03-09T09:09:26.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:26.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:26.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:27.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:27.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:27.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:27.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:28.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:28.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:28.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:28.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:29.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:29.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:29.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:29.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:29.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:29.582+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:30.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:30.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:30.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:30.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:31.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:31.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:31.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 rx=0x7f63e006ce60 tx=0x7f63f0027ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:31.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 secure :-1 s=READY pgs=3441 cs=0 l=1 rev1=1 rx=0x7f63e006ce60 tx=0x7f63f0027ed0).stop 2021-03-09T09:09:31.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:31.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:31.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:31.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023930 unknown :-1 s=CLOSED pgs=3441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:31.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:09:31.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:31.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:31.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:31.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:31.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:31.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:31.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:31.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027710 con 0x7f63e00070a0 2021-03-09T09:09:31.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:31.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f6400145bd0 2021-03-09T09:09:31.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:31.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012bd0 con 0x7f63e005c7c0 2021-03-09T09:09:31.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.114+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:31.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.115+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:31.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.115+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:31.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.115+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:31.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.115+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:31.124 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:31.115+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9160 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:31.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.116+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:31.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.116+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:09:31.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.116+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:31.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.116+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:31.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.116+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:31.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.116+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:31.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.116+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:31.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:31.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:31.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:31.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:31.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:31.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:09:31.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:09:31.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:31.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:31.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d4e0 con 0x7f63e005c7c0 2021-03-09T09:09:31.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:31.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:31.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:09:31.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:09:31.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:31.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c00 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 rx=0x7f63fc04a710 tx=0x7f63fc04a7b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:31.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:09:31.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.117+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:31.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:09:31.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.119+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:31.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.119+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:31.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.119+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:31.132 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:31.133 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:31.133 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:31.133 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:31.133 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:31.133 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:31.134 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:31.134 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:31.134 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:31.134 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:31.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0583e0 con 0x7f63e005c7c0 2021-03-09T09:09:31.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.119+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:31.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.119+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:09:31.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.120+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:09:31.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:31.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:31.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:32.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:32.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:32.497+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:33.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:33.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:33.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:33.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:34.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:34.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:34.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:34.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:34.583+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:35.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:35.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:35.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:35.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:36.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.116+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0015c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:09:36.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.116+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0015c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:36.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.116+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c00 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 rx=0x7f63fc04a710 tx=0x7f63fc04a7b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:36.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.116+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c00 secure :-1 s=READY pgs=3455 cs=0 l=1 rev1=1 rx=0x7f63fc04a710 tx=0x7f63fc04a7b0).stop 2021-03-09T09:09:36.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:36.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:36.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0015c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:36.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0015c00 unknown :-1 s=CLOSED pgs=3455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:36.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:09:36.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:36.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:36.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:36.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:36.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:36.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:36.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:36.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:09:36.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:36.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f6400145bd0 2021-03-09T09:09:36.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:36.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f6400146050 2021-03-09T09:09:36.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:36.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:36.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:36.125 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9199 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:36.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:36.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:36.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:36.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.118+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:09:36.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.118+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:36.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.118+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:36.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.118+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:36.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.118+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:36.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.118+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:36.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:36.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:36.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:36.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:36.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:36.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:09:36.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:09:36.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:36.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:36.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066290 con 0x7f6400146050 2021-03-09T09:09:36.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:36.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:36.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d4e0 con 0x7f6400146050 2021-03-09T09:09:36.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:09:36.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.120+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:36.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.121+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ed90 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 rx=0x7f63fc00f180 tx=0x7f63fc051f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:36.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.121+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:09:36.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.121+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:36.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0325e0 con 0x7f6400146050 2021-03-09T09:09:36.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.125+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:36.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.125+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:36.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.125+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:36.133 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:36.134 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:36.134 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:36.134 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:36.134 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:36.135 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:36.135 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:36.135 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:36.135 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:36.135 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:36.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:09:36.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.125+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:36.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.125+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:09:36.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f6400146050 2021-03-09T09:09:36.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:36.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:36.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:37.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:37.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:38.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:38.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:38.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:38.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:39.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:39.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:39.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:39.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:39.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:39.584+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:40.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:40.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:40.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:40.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:41.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:41.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:41.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ed90 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 rx=0x7f63fc00f180 tx=0x7f63fc051f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:41.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ed90 secure :-1 s=READY pgs=3470 cs=0 l=1 rev1=1 rx=0x7f63fc00f180 tx=0x7f63fc051f80).stop 2021-03-09T09:09:41.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:41.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:41.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002ed90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:41.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ed90 unknown :-1 s=CLOSED pgs=3470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:41.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:09:41.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:41.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:41.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:41.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:41.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:41.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c310 con 0x7f63e00070a0 2021-03-09T09:09:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027710 con 0x7f6400145bd0 2021-03-09T09:09:41.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:41.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f63e005c7c0 2021-03-09T09:09:41.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:41.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:41.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:41.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:41.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.118+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:41.127 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9230 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:41.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:41.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:09:41.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:41.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:41.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:41.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:41.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:41.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:41.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:41.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.119+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:41.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:41.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:41.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:09:41.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:09:41.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:41.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:41.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:09:41.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:41.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:41.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00282d0 con 0x7f63e005c7c0 2021-03-09T09:09:41.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:09:41.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:41.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023930 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f0013bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:41.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:09:41.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.120+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:41.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f63e005c7c0 2021-03-09T09:09:41.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.124+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:41.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.124+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:41.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.124+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:41.136 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:41.136 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:41.136 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:41.137 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:41.137 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:41.137 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:41.137 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:41.137 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:41.138 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:41.138 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:41.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e005c7c0 2021-03-09T09:09:41.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.125+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:41.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.125+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:09:41.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f63e005c7c0 2021-03-09T09:09:41.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:41.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:41.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:42.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:42.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:42.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:42.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:42.499+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:43.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:43.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:43.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:43.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:44.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:44.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:44.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:44.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:44.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:44.585+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:45.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:45.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:45.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:45.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:46.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:46.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:46.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023930 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f0013bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:46.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023930 secure :-1 s=READY pgs=3479 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f0013bf0).stop 2021-03-09T09:09:46.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:46.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:46.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0023930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:46.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0023930 unknown :-1 s=CLOSED pgs=3479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:46.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:09:46.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:46.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:46.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:46.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:46.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:46.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:46.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:46.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e00070a0 2021-03-09T09:09:46.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:46.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:09:46.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:46.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028320 con 0x7f6400146050 2021-03-09T09:09:46.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:46.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.120+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:46.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.120+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:46.129 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:46.120+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9274 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:46.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:09:46.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:46.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:46.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:46.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:09:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:09:46.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d640 con 0x7f6400146050 2021-03-09T09:09:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:46.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:46.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066290 con 0x7f6400146050 2021-03-09T09:09:46.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.121+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:09:46.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.122+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:46.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.123+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:09:46.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.123+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:46.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.123+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078030 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc065f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:46.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f6400146050 2021-03-09T09:09:46.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.125+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:46.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.125+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:46.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.125+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:46.137 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:46.137 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:46.137 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:46.137 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:46.137 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:46.138 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:46.138 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:46.138 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:46.138 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:46.139 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:46.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f6400146050 2021-03-09T09:09:46.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.126+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:46.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.127+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:09:46.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f6400146050 2021-03-09T09:09:46.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:46.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:46.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:47.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:47.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:47.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:47.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:48.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:48.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:48.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:48.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:49.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:49.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:49.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:49.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:49.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:49.586+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:50.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:50.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:50.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:50.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:51.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0078030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:09:51.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0078030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:51.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078030 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc065f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:51.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078030 secure :-1 s=READY pgs=3498 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc065f50).stop 2021-03-09T09:09:51.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:51.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:51.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0078030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:51.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0078030 unknown :-1 s=CLOSED pgs=3498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:51.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:09:51.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:51.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:51.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00139e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:51.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:51.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:51.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:51.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:51.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63e00070a0 2021-03-09T09:09:51.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:51.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f6400145bd0 2021-03-09T09:09:51.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:51.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:09:51.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:51.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:51.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:51.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:51.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.121+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:51.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.122+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00139e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:51.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.122+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:09:51.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.122+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:51.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.122+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:51.131 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:51.122+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9313 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:51.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.122+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:51.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.122+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:51.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.122+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:51.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:51.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:51.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:51.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:51.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:51.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:09:51.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:09:51.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:51.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:51.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:09:51.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:51.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:51.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d640 con 0x7f63e005c7c0 2021-03-09T09:09:51.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:09:51.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:51.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:09:51.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00139e0 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc066950).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:51.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.123+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:51.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00b5d0 con 0x7f63e005c7c0 2021-03-09T09:09:51.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.124+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:51.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.124+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:51.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.124+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:51.138 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:51.138 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:51.139 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:51.139 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:51.139 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:51.139 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:51.140 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:51.140 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:51.140 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:51.140 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:51.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f63e005c7c0 2021-03-09T09:09:51.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.124+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:51.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.124+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:09:51.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63e005c7c0 2021-03-09T09:09:51.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:51.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:51.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:52.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:52.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:52.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:52.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:52.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:52.500+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:53.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:53.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:53.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:53.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:54.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:54.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:54.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:54.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:54.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:54.587+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:09:55.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:55.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:55.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:55.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:56.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.122+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00139e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:09:56.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.122+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00139e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:09:56.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.122+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00139e0 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc066950).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:09:56.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.122+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00139e0 secure :-1 s=READY pgs=3504 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc066950).stop 2021-03-09T09:09:56.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.122+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:56.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.122+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:09:56.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.122+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00139e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:09:56.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.122+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00139e0 unknown :-1 s=CLOSED pgs=3504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:09:56.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:09:56.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:56.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:09:56.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:56.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:09:56.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:09:56.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:09:56.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:56.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f63e00070a0 2021-03-09T09:09:56.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:56.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028320 con 0x7f6400145bd0 2021-03-09T09:09:56.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:09:56.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077840 con 0x7f6400146050 2021-03-09T09:09:56.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:09:56.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:09:56.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:56.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:09:56.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:09:56.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:09:56.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:09:56.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:09:56.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:09:56.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:09:56.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:09:56.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.123+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:09:56.133 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9341 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:09:56.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:09:56.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:56.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:56.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:09:56.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:09:56.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:09:56.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:09:56.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:09:56.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:56.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:09:56.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:09:56.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:09:56.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a8d0 con 0x7f6400146050 2021-03-09T09:09:56.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:09:56.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.124+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:56.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.125+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4005280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:09:56.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.125+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:09:56.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.125+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:09:56.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40035e0 con 0x7f6400146050 2021-03-09T09:09:56.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.126+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:09:56.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.126+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:09:56.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.126+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:09:56.139 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:09:56.140 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:09:56.140 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:56.140 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:09:56.140 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:09:56.141 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:09:56.141 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:09:56.141 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:09:56.141 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:09:56.141 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:09:56.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40035e0 con 0x7f6400146050 2021-03-09T09:09:56.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.127+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:09:56.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.127+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:09:56.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f6400146050 2021-03-09T09:09:56.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:56.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:56.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:57.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:57.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:57.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:57.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:58.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:58.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:58.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:58.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:59.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:59.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:09:59.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:59.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:09:59.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:09:59.588+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:00.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:00.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:00.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:00.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:01.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:10:01.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:01.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4005280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:01.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3518 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4005280).stop 2021-03-09T09:10:01.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:01.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:01.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:01.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=CLOSED pgs=3518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:01.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:10:01.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:01.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:01.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:01.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:01.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:01.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:01.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:01.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e00070a0 2021-03-09T09:10:01.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:01.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f6400145bd0 2021-03-09T09:10:01.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:01.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f63e005c7c0 2021-03-09T09:10:01.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.124+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:01.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.125+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:01.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.125+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:01.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.125+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:01.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.125+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:10:01.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.125+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:01.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.125+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:01.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.125+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:01.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.125+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:01.135 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9387 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:01.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:01.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:01.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:01.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:01.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:01.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:01.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:01.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:01.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:10:01.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:10:01.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:01.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:01.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019c00 con 0x7f63e005c7c0 2021-03-09T09:10:01.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:01.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:01.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:10:01.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:10:01.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.126+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:01.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.127+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:10:01.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.127+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:01.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.127+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b570 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 rx=0x7f63e00097d0 tx=0x7f63f0013f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:01.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e005c7c0 2021-03-09T09:10:01.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.129+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:01.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.129+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:01.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.129+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:01.143 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:01.143 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:01.143 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:01.144 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:01.144 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:01.144 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:01.144 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:01.144 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:01.145 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:01.145 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:01.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d290 con 0x7f63e005c7c0 2021-03-09T09:10:01.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.129+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:01.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.129+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:10:01.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0054030 con 0x7f63e005c7c0 2021-03-09T09:10:01.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:01.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:01.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:02.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:02.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:02.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:02.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:02.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:02.501+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:03.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:03.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:03.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:03.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:04.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:04.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:04.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:04.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:04.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:04.589+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:05.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:05.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:05.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:05.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b570 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 rx=0x7f63e00097d0 tx=0x7f63f0013f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:06.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b570 secure :-1 s=READY pgs=3534 cs=0 l=1 rev1=1 rx=0x7f63e00097d0 tx=0x7f63f0013f00).stop 2021-03-09T09:10:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001b570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:06.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001b570 unknown :-1 s=CLOSED pgs=3534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:06.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:10:06.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:06.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:06.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:06.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:06.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:06.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:06.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:06.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bc00 con 0x7f63e00070a0 2021-03-09T09:10:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027710 con 0x7f6400145bd0 2021-03-09T09:10:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022fc0 con 0x7f6400146050 2021-03-09T09:10:06.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:06.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.126+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:06.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.127+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:06.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.127+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:10:06.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.127+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:06.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.127+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:06.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.127+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:06.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.127+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:06.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.127+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:06.137 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:06.127+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9417 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:06.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.127+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:06.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:06.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:06.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:06.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:06.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:10:06.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:10:06.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a8d0 con 0x7f6400146050 2021-03-09T09:10:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:06.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:10:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:10:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:10:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:06.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.128+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 rx=0x7f63f4061770 tx=0x7f63f40613f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4047810 con 0x7f6400146050 2021-03-09T09:10:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.130+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.130+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:06.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.130+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:06.143 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:06.144 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:06.144 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:06.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.136+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4047810 con 0x7f6400146050 2021-03-09T09:10:06.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.136+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:06.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.136+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:10:06.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.136+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:10:06.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:06.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:06.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:07.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:07.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:07.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:07.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:08.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:08.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:08.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:08.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:09.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:09.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:09.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:09.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:09.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:09.590+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:10.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:10.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:10.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:10.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:11.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:10:11.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:11.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 rx=0x7f63f4061770 tx=0x7f63f40613f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:11.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3545 cs=0 l=1 rev1=1 rx=0x7f63f4061770 tx=0x7f63f40613f0).stop 2021-03-09T09:10:11.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:11.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:11.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:11.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=CLOSED pgs=3545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:11.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:10:11.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00273d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:11.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:11.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:11.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:11.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:11.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:11.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:11.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c010 con 0x7f63e00070a0 2021-03-09T09:10:11.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:11.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:10:11.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:11.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63e005c7c0 2021-03-09T09:10:11.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:11.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:11.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:11.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00273d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:11.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:10:11.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:11.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:11.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:11.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:11.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:11.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:11.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.128+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:11.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:11.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:11.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:11.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:11.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:11.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:10:11.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:10:11.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:11.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:11.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c2c0 con 0x7f63e005c7c0 2021-03-09T09:10:11.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:11.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:11.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a8d0 con 0x7f63e005c7c0 2021-03-09T09:10:11.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:10:11.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:11.142 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9454 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:11.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00273d0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 rx=0x7f63f4047980 tx=0x7f63f401af50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:11.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:10:11.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.129+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:11.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402aca0 con 0x7f63e005c7c0 2021-03-09T09:10:11.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.130+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:11.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.130+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:11.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.130+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:11.145 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:11.145 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:11.146 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:11.146 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:11.146 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:11.146 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:11.146 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:11.147 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:11.147 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:11.147 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:11.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63e005c7c0 2021-03-09T09:10:11.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.130+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:11.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.130+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:10:11.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.132+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4062560 con 0x7f63e005c7c0 2021-03-09T09:10:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:11.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:12.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:12.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:12.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:12.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:12.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:12.503+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:13.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:13.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:13.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:13.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:14.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:14.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:14.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:14.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:14.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:14.591+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:15.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:15.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:15.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:15.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:16.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00273d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:10:16.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00273d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:16.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00273d0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 rx=0x7f63f4047980 tx=0x7f63f401af50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:16.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00273d0 secure :-1 s=READY pgs=3559 cs=0 l=1 rev1=1 rx=0x7f63f4047980 tx=0x7f63f401af50).stop 2021-03-09T09:10:16.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:16.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:16.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00273d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:16.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00273d0 unknown :-1 s=CLOSED pgs=3559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:16.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:10:16.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:16.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:16.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:16.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:16.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:16.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:16.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:16.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f63e00070a0 2021-03-09T09:10:16.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:16.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077840 con 0x7f6400145bd0 2021-03-09T09:10:16.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:16.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022fc0 con 0x7f6400146050 2021-03-09T09:10:16.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.129+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:16.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:16.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:16.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:16.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:16.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:16.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:10:16.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:16.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:16.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:16.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:16.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.130+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:16.142 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9493 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:16.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:16.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:16.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:16.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:16.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:16.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:10:16.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:10:16.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:16.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:16.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:10:16.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:16.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:16.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:10:16.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:10:16.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.131+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:16.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.132+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:16.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.132+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:10:16.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.132+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:16.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.133+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00b5d0 con 0x7f6400146050 2021-03-09T09:10:16.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.133+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:16.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.133+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:16.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.133+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:16.149 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:16.149 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:16.149 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:16.150 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:16.150 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:16.150 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:16.150 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:16.151 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:16.151 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:16.151 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:16.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:10:16.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.134+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:16.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.134+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:10:16.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f6400146050 2021-03-09T09:10:16.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:16.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:16.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:17.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:17.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:17.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:17.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:18.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:18.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:18.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:18.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:19.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:19.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:19.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:19.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:19.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:19.592+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:20.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:20.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:20.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:20.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:21.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:21.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:21.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:21.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3573 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc009990).stop 2021-03-09T09:10:21.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:21.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:21.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:21.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=CLOSED pgs=3573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:21.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:10:21.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:21.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:21.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:21.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:21.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:21.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:21.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:21.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63e00070a0 2021-03-09T09:10:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c010 con 0x7f6400145bd0 2021-03-09T09:10:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:10:21.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:21.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:21.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:21.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:21.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.131+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:21.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.132+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:21.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.132+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:10:21.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.132+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:21.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.132+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:21.141 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:21.132+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9538 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:21.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:21.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:21.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:21.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:21.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:21.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:21.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:21.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:21.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:10:21.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:10:21.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:21.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:21.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a8d0 con 0x7f63e005c7c0 2021-03-09T09:10:21.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:21.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:21.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a60 con 0x7f63e005c7c0 2021-03-09T09:10:21.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.134+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:10:21.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.134+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:21.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.134+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:10:21.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.134+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:21.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.134+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027e20 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc04f4f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:21.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04f000 con 0x7f63e005c7c0 2021-03-09T09:10:21.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.135+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:21.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.135+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:21.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.135+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:21.149 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:21.150 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:21.150 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:21.150 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:21.150 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:21.151 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:21.151 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:21.151 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:21.151 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:21.152 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:21.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0583e0 con 0x7f63e005c7c0 2021-03-09T09:10:21.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.137+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:21.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.137+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:10:21.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04f000 con 0x7f63e005c7c0 2021-03-09T09:10:21.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:21.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:21.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:22.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:22.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:22.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:22.504+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:23.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:23.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:23.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:23.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:24.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:24.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:24.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:24.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:24.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:24.593+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:25.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:25.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:25.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:25.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:26.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0027e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:26.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0027e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:26.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027e20 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc04f4f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:26.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027e20 secure :-1 s=READY pgs=3590 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc04f4f0).stop 2021-03-09T09:10:26.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:26.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:26.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0027e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:26.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027e20 unknown :-1 s=CLOSED pgs=3590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:26.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:10:26.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:26.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:26.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:26.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:26.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:26.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:26.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:26.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e00070a0 2021-03-09T09:10:26.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:26.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f6400145bd0 2021-03-09T09:10:26.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:26.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065550 con 0x7f6400146050 2021-03-09T09:10:26.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:26.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:26.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:26.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:26.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.133+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:26.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.134+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:26.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.134+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:10:26.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.134+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:26.143 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:26.134+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9575 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:26.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.134+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:26.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.134+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:26.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.134+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.134+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:26.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:26.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:10:26.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:10:26.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:26.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:26.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000bbb0 con 0x7f6400146050 2021-03-09T09:10:26.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:26.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:26.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019c00 con 0x7f6400146050 2021-03-09T09:10:26.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:10:26.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.135+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:26.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.136+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 rx=0x7f63e002ed40 tx=0x7f63f0055eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:26.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.136+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:10:26.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.136+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0054030 con 0x7f6400146050 2021-03-09T09:10:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.137+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.137+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.137+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:26.151 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:26.151 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:26.151 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:26.151 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:26.152 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:26.152 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:26.152 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:26.152 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:26.152 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:26.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.140+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d290 con 0x7f6400146050 2021-03-09T09:10:26.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.140+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:26.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.140+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:10:26.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0054030 con 0x7f6400146050 2021-03-09T09:10:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:26.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:27.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:27.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:27.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:27.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:28.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:28.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:28.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:28.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:29.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:29.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:29.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:29.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:29.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:29.594+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:30.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:30.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:30.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:30.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:31.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:10:31.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:31.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 rx=0x7f63e002ed40 tx=0x7f63f0055eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:31.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3607 cs=0 l=1 rev1=1 rx=0x7f63e002ed40 tx=0x7f63f0055eb0).stop 2021-03-09T09:10:31.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:31.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:31.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:31.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=CLOSED pgs=3607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:31.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:10:31.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:31.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:31.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:31.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:31.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:31.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:31.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:31.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014810 con 0x7f63e00070a0 2021-03-09T09:10:31.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:31.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022fc0 con 0x7f6400145bd0 2021-03-09T09:10:31.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:31.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:10:31.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:31.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:31.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:31.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:31.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.135+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:31.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.136+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:31.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.136+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:10:31.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.136+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:31.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.136+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:31.145 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:31.136+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9608 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:31.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:31.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:31.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:31.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:31.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:31.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:31.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:31.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:31.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:10:31.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:10:31.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:31.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:31.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:10:31.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:31.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:31.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a8d0 con 0x7f63e005c7c0 2021-03-09T09:10:31.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:10:31.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.137+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:31.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.138+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c840 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63fc04f560).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:31.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.138+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:10:31.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.138+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:31.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f63e005c7c0 2021-03-09T09:10:31.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.139+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:31.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.139+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:31.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.139+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:31.153 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:31.153 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:31.153 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:31.154 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:31.154 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:31.154 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:31.154 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:31.155 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:31.155 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:31.155 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:31.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.140+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f63e005c7c0 2021-03-09T09:10:31.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.140+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:31.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.140+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:10:31.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f63e005c7c0 2021-03-09T09:10:31.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:31.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:31.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:32.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:32.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:32.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:32.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:32.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:32.506+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:33.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:33.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:33.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:33.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:34.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:34.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:34.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:34.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:34.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:34.595+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:35.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:35.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:35.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:35.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:36.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:10:36.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:36.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c840 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63fc04f560).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:36.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c840 secure :-1 s=READY pgs=3617 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63fc04f560).stop 2021-03-09T09:10:36.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:36.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:36.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:36.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c840 unknown :-1 s=CLOSED pgs=3617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:36.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:10:36.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:36.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:36.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:36.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:36.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:36.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:36.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:36.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63e00070a0 2021-03-09T09:10:36.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:36.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c010 con 0x7f6400145bd0 2021-03-09T09:10:36.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:36.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f6400146050 2021-03-09T09:10:36.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:36.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:36.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:36.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:36.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:36.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:36.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:10:36.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:36.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.137+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:36.147 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:36.138+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9644 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:36.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.138+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:36.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.138+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:36.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.138+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:36.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:36.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:36.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:36.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:36.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:36.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:10:36.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:10:36.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:36.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:36.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:10:36.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:36.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:36.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:10:36.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:10:36.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:36.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:10:36.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:36.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.139+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc066300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:36.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.140+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04f250 con 0x7f6400146050 2021-03-09T09:10:36.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.140+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.140+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.140+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:36.155 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:36.155 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:36.155 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:36.155 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:36.155 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:36.156 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:36.156 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:36.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04f250 con 0x7f6400146050 2021-03-09T09:10:36.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.142+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:36.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.142+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:10:36.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f6400146050 2021-03-09T09:10:36.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:36.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:36.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:37.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:37.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:37.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:38.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:38.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:38.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:38.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:39.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:39.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:39.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:39.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:39.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:39.596+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:40.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:40.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:40.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:40.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:41.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:41.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:41.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc066300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:41.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3630 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc066300).stop 2021-03-09T09:10:41.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:41.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:41.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:41.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=CLOSED pgs=3630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:41.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:10:41.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:41.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:41.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:41.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:41.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:41.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:41.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:41.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f63e00070a0 2021-03-09T09:10:41.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:41.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014810 con 0x7f6400145bd0 2021-03-09T09:10:41.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:41.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028520 con 0x7f63e005c7c0 2021-03-09T09:10:41.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.138+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:41.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.139+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:41.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.139+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:41.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.139+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:41.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.139+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:41.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.139+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:41.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.139+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:10:41.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.139+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:41.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.139+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:41.148 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:41.139+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9689 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:41.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:41.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:41.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:41.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:41.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:41.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:41.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:41.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:41.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:10:41.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:10:41.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:41.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:41.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025b20 con 0x7f63e005c7c0 2021-03-09T09:10:41.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:41.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:41.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c2c0 con 0x7f63e005c7c0 2021-03-09T09:10:41.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:10:41.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.140+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:41.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006d000 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 rx=0x7f63f4063340 tx=0x7f63f40035e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:41.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:10:41.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:41.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f63e005c7c0 2021-03-09T09:10:41.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:41.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:41.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:41.155 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:41.156 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:41.156 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:41.156 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:41.156 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:41.156 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:41.157 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:41.157 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:41.157 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:41.157 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:41.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63e005c7c0 2021-03-09T09:10:41.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:41.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.141+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:10:41.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402aca0 con 0x7f63e005c7c0 2021-03-09T09:10:41.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:41.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:41.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:42.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:42.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:42.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:42.507+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:43.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:43.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:43.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:43.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:44.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:44.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:44.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:44.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:44.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:44.597+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:45.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:45.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:45.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:45.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:46.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:10:46.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:46.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006d000 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 rx=0x7f63f4063340 tx=0x7f63f40035e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:46.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006d000 secure :-1 s=READY pgs=3643 cs=0 l=1 rev1=1 rx=0x7f63f4063340 tx=0x7f63f40035e0).stop 2021-03-09T09:10:46.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:46.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:46.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006d000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:46.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006d000 unknown :-1 s=CLOSED pgs=3643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:46.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:10:46.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:46.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:46.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:46.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:46.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:46.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:46.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:46.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:10:46.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:46.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cfa0 con 0x7f6400145bd0 2021-03-09T09:10:46.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:46.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:10:46.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:46.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:46.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:46.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:46.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:46.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:46.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:10:46.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:46.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.140+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:46.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.141+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:46.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.141+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:46.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.141+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:46.157 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:46.142+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9724 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:46.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.142+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:46.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.142+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:46.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:46.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:46.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:46.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:10:46.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:10:46.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:46.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013dc0 con 0x7f6400146050 2021-03-09T09:10:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:10:46.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:10:46.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:46.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc02c6a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:46.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:10:46.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.143+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:46.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c220 con 0x7f6400146050 2021-03-09T09:10:46.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.144+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:46.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.144+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:46.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.144+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:46.163 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:46.164 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:46.164 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:46.164 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:46.164 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:46.164 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:46.165 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:46.165 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:46.165 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:46.165 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:46.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f6400146050 2021-03-09T09:10:46.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.144+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:46.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.144+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:10:46.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f6400146050 2021-03-09T09:10:46.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:46.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:46.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:47.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:47.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:47.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:47.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:48.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:48.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:48.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:48.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:49.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:49.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:49.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:49.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:49.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:49.598+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:50.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:50.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:50.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:50.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:51.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:51.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:51.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc02c6a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:51.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3659 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc02c6a0).stop 2021-03-09T09:10:51.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:51.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:51.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:51.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=CLOSED pgs=3659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:51.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:10:51.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00166b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:51.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:51.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:51.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:51.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:51.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:51.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:51.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63e00070a0 2021-03-09T09:10:51.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:51.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400145bd0 2021-03-09T09:10:51.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:51.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e005c7c0 2021-03-09T09:10:51.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:51.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:51.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.142+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:51.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:51.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:51.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00166b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:51.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:10:51.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:51.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:51.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:51.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:51.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:51.154 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:51.143+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9757 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:51.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:51.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:51.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:10:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:10:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066290 con 0x7f63e005c7c0 2021-03-09T09:10:51.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:51.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:51.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025b20 con 0x7f63e005c7c0 2021-03-09T09:10:51.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:10:51.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.144+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:51.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.145+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:10:51.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.145+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:51.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.145+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00166b0 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 rx=0x7f63f4023960 tx=0x7f63f4017f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:51.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034a30 con 0x7f63e005c7c0 2021-03-09T09:10:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.146+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.146+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.146+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:51.160 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:51.161 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:51.162 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:51.162 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:51.162 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:51.162 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:51.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.148+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034a30 con 0x7f63e005c7c0 2021-03-09T09:10:51.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.148+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:51.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.148+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:10:51.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402ac30 con 0x7f63e005c7c0 2021-03-09T09:10:51.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:51.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:51.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:52.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:52.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:52.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:52.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:52.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:52.508+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:53.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:53.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:53.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:53.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:54.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:54.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:54.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:54.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:54.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:54.599+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:10:55.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:55.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:55.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:55.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:56.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00166b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:10:56.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00166b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:10:56.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00166b0 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 rx=0x7f63f4023960 tx=0x7f63f4017f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:10:56.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00166b0 secure :-1 s=READY pgs=3670 cs=0 l=1 rev1=1 rx=0x7f63f4023960 tx=0x7f63f4017f80).stop 2021-03-09T09:10:56.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:56.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:10:56.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00166b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:10:56.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00166b0 unknown :-1 s=CLOSED pgs=3670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:10:56.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:10:56.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:56.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:10:56.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:56.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:10:56.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:10:56.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:10:56.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:56.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:10:56.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:56.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cfa0 con 0x7f6400145bd0 2021-03-09T09:10:56.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:10:56.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b870 con 0x7f6400146050 2021-03-09T09:10:56.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:10:56.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:10:56.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:56.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:10:56.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:10:56.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.144+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:10:56.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:10:56.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:10:56.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:10:56.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:10:56.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:10:56.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:10:56.155 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9798 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:10:56.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:10:56.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:56.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.145+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:56.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:10:56.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:10:56.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:10:56.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:10:56.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:10:56.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:56.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f7f0 con 0x7f6400146050 2021-03-09T09:10:56.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:10:56.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:10:56.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000bbb0 con 0x7f6400146050 2021-03-09T09:10:56.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:10:56.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:56.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 rx=0x7f63f001b270 tx=0x7f63f00487e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:10:56.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:10:56.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.146+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:10:56.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0054260 con 0x7f6400146050 2021-03-09T09:10:56.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.147+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:10:56.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.147+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:10:56.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.147+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:10:56.161 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:10:56.161 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:10:56.161 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:56.162 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:10:56.162 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:10:56.162 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:10:56.162 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:10:56.163 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:10:56.163 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:10:56.163 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:10:56.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.148+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0054260 con 0x7f6400146050 2021-03-09T09:10:56.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.148+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:10:56.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.148+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:10:56.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f6400146050 2021-03-09T09:10:56.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:56.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:56.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:57.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:57.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:57.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:57.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:58.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:58.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:58.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:58.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:59.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:59.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:10:59.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:59.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:10:59.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:10:59.600+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:00.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:00.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:00.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:00.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:01.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:11:01.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:01.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 rx=0x7f63f001b270 tx=0x7f63f00487e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:01.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 secure :-1 s=READY pgs=3685 cs=0 l=1 rev1=1 rx=0x7f63f001b270 tx=0x7f63f00487e0).stop 2021-03-09T09:11:01.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:01.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:01.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007bb10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:01.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007bb10 unknown :-1 s=CLOSED pgs=3685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:01.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:11:01.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0025040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:01.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:01.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:01.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:01.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:01.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:01.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:01.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00791b0 con 0x7f63e00070a0 2021-03-09T09:11:01.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:01.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f6400145bd0 2021-03-09T09:11:01.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:01.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63e005c7c0 2021-03-09T09:11:01.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:01.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:01.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:01.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:01.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:01.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0025040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:01.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:11:01.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:01.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:01.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:01.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:01.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.146+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:01.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:01.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:01.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:01.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:01.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:01.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:11:01.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:11:01.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:01.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:01.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a60 con 0x7f63e005c7c0 2021-03-09T09:11:01.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:01.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:01.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066290 con 0x7f63e005c7c0 2021-03-09T09:11:01.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:11:01.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:01.161 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:01.147+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9830 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:01.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.148+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:11:01.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.148+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:01.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.148+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0025040 secure :-1 s=READY pgs=3696 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40237c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:01.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048380 con 0x7f63e005c7c0 2021-03-09T09:11:01.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.150+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:01.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.150+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:01.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.150+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:01.163 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:01.164 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:01.164 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:01.164 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:01.164 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:01.164 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:01.165 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:01.165 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:01.165 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:01.165 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:01.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40487b0 con 0x7f63e005c7c0 2021-03-09T09:11:01.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.150+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:01.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.150+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:11:01.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:11:01.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:01.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:01.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:02.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:02.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:02.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:02.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:02.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:02.510+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:03.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:03.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:03.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:03.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:04.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:04.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:04.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:04.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:04.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:04.601+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:05.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:05.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:05.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:05.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:06.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0025040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:11:06.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0025040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:06.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0025040 secure :-1 s=READY pgs=3696 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40237c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:06.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0025040 secure :-1 s=READY pgs=3696 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40237c0).stop 2021-03-09T09:11:06.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:06.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:06.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0025040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:06.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0025040 unknown :-1 s=CLOSED pgs=3696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:06.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:11:06.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:06.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:06.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:06.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:06.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:06.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:06.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016040 con 0x7f63e00070a0 2021-03-09T09:11:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024210 con 0x7f6400145bd0 2021-03-09T09:11:06.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:11:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:06.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.147+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:06.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:11:06.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:06.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:06.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:06.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:06.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:06.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:06.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:11:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:11:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:06.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:06.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:11:06.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:06.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:06.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:11:06.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:11:06.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.148+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:06.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404db90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:06.163 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9875 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:06.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:11:06.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:06.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:11:06.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:06.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:06.165 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:06.166 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:06.167 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:06.167 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:06.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404dd40 con 0x7f6400146050 2021-03-09T09:11:06.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:06.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.149+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:11:06.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.151+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:11:06.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:06.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:06.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:07.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:07.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:07.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:07.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:08.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:08.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:08.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:08.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:09.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:09.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:09.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:09.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:09.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:09.602+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:10.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:10.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:11.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:11:11.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:11.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404db90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:11.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 secure :-1 s=READY pgs=3709 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404db90).stop 2021-03-09T09:11:11.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:11.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:11.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007baf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:11.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 unknown :-1 s=CLOSED pgs=3709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:11.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:11:11.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:11.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:11.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:11.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:11.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:11.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:11.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:11.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e00070a0 2021-03-09T09:11:11.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:11.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:11:11.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:11.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029290 con 0x7f63e005c7c0 2021-03-09T09:11:11.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:11.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:11.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:11.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:11.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:11.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:11.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:11:11.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:11.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.150+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:11.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.151+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:11.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.151+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:11.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.151+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:11.160 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:11.151+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9908 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:11.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:11.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:11.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:11.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:11.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:11.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:11:11.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:11:11.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:11.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:11.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006a8b0 con 0x7f63e005c7c0 2021-03-09T09:11:11.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:11.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:11.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:11:11.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:11:11.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.152+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:11.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.153+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:11:11.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.153+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:11.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.153+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=3729 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f403d170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:11.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.154+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f403d5d0 con 0x7f63e005c7c0 2021-03-09T09:11:11.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.154+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:11.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.154+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:11.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.154+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:11.166 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:11.167 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:11.167 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:11.167 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:11.167 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:11.167 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:11.168 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:11.168 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:11.168 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:11.168 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:11.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404dd40 con 0x7f63e005c7c0 2021-03-09T09:11:11.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.155+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:11.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.155+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:11:11.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63e005c7c0 2021-03-09T09:11:11.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:11.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:11.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:12.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:12.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:12.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:12.511+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:13.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:13.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:13.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:13.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:14.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:14.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:14.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:14.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:14.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:14.603+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:15.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:15.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:15.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:15.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:16.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:11:16.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:16.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=3729 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f403d170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:16.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=3729 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f403d170).stop 2021-03-09T09:11:16.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:16.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:16.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:16.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=CLOSED pgs=3729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:16.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:11:16.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:16.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:16.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:16.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:16.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:16.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:16.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:16.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077840 con 0x7f63e00070a0 2021-03-09T09:11:16.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:16.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:11:16.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:16.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e9c0 con 0x7f6400146050 2021-03-09T09:11:16.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:16.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:16.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:16.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:16.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.151+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:16.160 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:16.152+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9950 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:16.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.152+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:16.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.152+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:11:16.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.152+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:16.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.152+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:16.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:16.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:16.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:16.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:16.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:16.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:16.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:16.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:16.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:11:16.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:11:16.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:16.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:16.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026670 con 0x7f6400146050 2021-03-09T09:11:16.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:16.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:16.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.153+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006a8b0 con 0x7f6400146050 2021-03-09T09:11:16.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.154+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:11:16.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.154+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:16.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.154+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4029fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:16.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.154+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:11:16.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.154+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:16.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4016c60 con 0x7f6400146050 2021-03-09T09:11:16.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.155+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:16.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.155+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:16.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.155+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:16.168 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:16.168 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:16.169 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:16.169 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:16.169 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:16.169 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:16.169 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:16.170 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:16.170 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:16.170 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:16.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4016a30 con 0x7f6400146050 2021-03-09T09:11:16.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.156+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:16.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.156+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:11:16.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f6400146050 2021-03-09T09:11:16.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:16.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:16.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:17.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:17.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:17.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:17.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:18.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:18.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:18.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:18.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:19.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:19.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:19.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:19.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:19.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:19.604+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:20.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:20.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:20.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:20.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:21.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:11:21.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:21.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4029fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:21.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4029fb0).stop 2021-03-09T09:11:21.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:21.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:21.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007baf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:21.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 unknown :-1 s=CLOSED pgs=3743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:21.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:11:21.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:21.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:21.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:21.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:21.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:21.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:21.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:21.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066400 con 0x7f63e00070a0 2021-03-09T09:11:21.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:21.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f6400145bd0 2021-03-09T09:11:21.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:21.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c670 con 0x7f63e005c7c0 2021-03-09T09:11:21.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:21.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:21.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.153+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:21.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:21.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:21.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:21.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:11:21.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:21.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:21.164 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 9988 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:21.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:21.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.154+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:21.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:21.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:21.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:11:21.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:11:21.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:21.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:21.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a60 con 0x7f63e005c7c0 2021-03-09T09:11:21.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:21.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:21.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f7f0 con 0x7f63e005c7c0 2021-03-09T09:11:21.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:11:21.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.155+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:21.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.156+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b890 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 rx=0x7f63f00655e0 tx=0x7f63f00657a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:21.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.156+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:11:21.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.156+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:21.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0065450 con 0x7f63e005c7c0 2021-03-09T09:11:21.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.157+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:21.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.157+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:21.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.157+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:21.172 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:21.172 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:21.172 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:21.172 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:21.173 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:21.173 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:21.173 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:21.173 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:21.174 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:21.174 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:21.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d340 con 0x7f63e005c7c0 2021-03-09T09:11:21.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.157+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:21.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.157+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:11:21.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0054040 con 0x7f63e005c7c0 2021-03-09T09:11:21.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:21.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:21.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:22.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:22.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:22.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:22.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:22.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:22.512+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:23.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:23.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:23.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:23.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:24.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:24.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:24.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:24.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:24.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:24.605+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:25.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:25.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:25.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:25.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:26.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:11:26.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:26.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b890 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 rx=0x7f63f00655e0 tx=0x7f63f00657a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:26.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b890 secure :-1 s=READY pgs=3753 cs=0 l=1 rev1=1 rx=0x7f63f00655e0 tx=0x7f63f00657a0).stop 2021-03-09T09:11:26.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:26.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:26.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:26.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b890 unknown :-1 s=CLOSED pgs=3753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:26.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:11:26.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:26.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:26.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:26.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:26.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:26.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:26.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:26.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014810 con 0x7f63e00070a0 2021-03-09T09:11:26.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:26.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e920 con 0x7f6400145bd0 2021-03-09T09:11:26.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:26.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:11:26.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:26.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:26.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:26.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:26.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:26.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:26.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:11:26.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:26.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:26.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:26.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:26.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.155+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:26.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:26.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:26.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:26.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:26.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:26.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:11:26.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:11:26.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:26.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:26.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:11:26.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:26.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:26.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026670 con 0x7f6400146050 2021-03-09T09:11:26.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:11:26.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:26.170 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:26.156+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10022 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:26.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.157+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:11:26.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.157+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:26.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.157+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 secure :-1 s=READY pgs=3765 cs=0 l=1 rev1=1 rx=0x7f63f4016e20 tx=0x7f63f401bf00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:26.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404bc40 con 0x7f6400146050 2021-03-09T09:11:26.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.158+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:26.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.158+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:26.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.158+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:26.172 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:26.173 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:26.173 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:26.174 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:26.174 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:26.174 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:26.174 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:26.175 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:26.175 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:26.175 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:26.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:11:26.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.158+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:26.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.158+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:11:26.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.159+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:11:26.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:26.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:26.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:27.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:27.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:27.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:27.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:28.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:28.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:28.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:28.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:29.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:29.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:29.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:29.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:29.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:29.606+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:30.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:30.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:30.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:30.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:31.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:11:31.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:31.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 secure :-1 s=READY pgs=3765 cs=0 l=1 rev1=1 rx=0x7f63f4016e20 tx=0x7f63f401bf00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:31.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 secure :-1 s=READY pgs=3765 cs=0 l=1 rev1=1 rx=0x7f63f4016e20 tx=0x7f63f401bf00).stop 2021-03-09T09:11:31.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:31.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:31.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007baf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:31.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007baf0 unknown :-1 s=CLOSED pgs=3765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:31.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:11:31.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:31.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:31.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:31.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:31.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:31.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:31.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:31.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:11:31.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:31.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016040 con 0x7f6400145bd0 2021-03-09T09:11:31.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:31.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f63e005c7c0 2021-03-09T09:11:31.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:31.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:31.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:31.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:31.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:31.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:31.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:11:31.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:31.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:31.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.157+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:31.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:31.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:31.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:31.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:31.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:31.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:31.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:31.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:11:31.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:11:31.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:31.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:31.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63e005c7c0 2021-03-09T09:11:31.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:31.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:31.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a60 con 0x7f63e005c7c0 2021-03-09T09:11:31.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:11:31.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:31.171 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:31.158+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10064 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:31.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.159+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:11:31.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.159+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 secure :-1 s=READY pgs=3781 cs=0 l=1 rev1=1 rx=0x7f63f00407e0 tx=0x7f63f0055f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:31.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.159+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:31.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.161+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63e005c7c0 2021-03-09T09:11:31.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.161+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:31.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.161+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:31.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.161+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:31.173 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:31.173 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:31.174 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:31.174 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:31.174 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:31.174 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:31.174 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:31.175 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:31.175 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:31.175 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:31.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63e005c7c0 2021-03-09T09:11:31.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.165+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:31.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.165+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:11:31.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d300 con 0x7f63e005c7c0 2021-03-09T09:11:31.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:31.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:31.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:32.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:32.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:32.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:32.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:32.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:32.514+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:33.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:33.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:33.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:33.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:34.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:34.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:34.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:34.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:34.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:34.607+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:35.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:35.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:35.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:35.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:36.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0017d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:11:36.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0017d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:36.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 secure :-1 s=READY pgs=3781 cs=0 l=1 rev1=1 rx=0x7f63f00407e0 tx=0x7f63f0055f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:36.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 secure :-1 s=READY pgs=3781 cs=0 l=1 rev1=1 rx=0x7f63f00407e0 tx=0x7f63f0055f90).stop 2021-03-09T09:11:36.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:36.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:36.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0017d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:36.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0017d40 unknown :-1 s=CLOSED pgs=3781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:36.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:11:36.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:36.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:36.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:36.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:36.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:36.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:36.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:36.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022b30 con 0x7f63e00070a0 2021-03-09T09:11:36.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:36.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d570 con 0x7f6400145bd0 2021-03-09T09:11:36.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:36.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:11:36.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:36.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:36.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:36.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:36.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:36.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:36.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:11:36.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.159+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:36.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:36.174 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10100 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:36.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:11:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:11:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:36.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.160+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.161+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:11:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.161+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.161+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:36.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.161+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T09:11:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.161+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:11:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.161+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.161+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 secure :-1 s=READY pgs=3791 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f000eef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:36.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.161+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:11:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.161+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027e70 con 0x7f6400146050 2021-03-09T09:11:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.162+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:36.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.162+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.162+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:36.181 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:36.182 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:36.182 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:36.182 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:36.182 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:36.183 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:36.183 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:36.183 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:36.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027e70 con 0x7f6400146050 2021-03-09T09:11:36.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.163+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:36.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.163+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:11:36.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f6400146050 2021-03-09T09:11:36.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:36.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:36.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:37.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:37.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:37.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:38.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:38.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:38.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:38.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:39.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:39.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:39.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:39.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:39.608+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:40.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:40.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:40.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:40.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:41.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:11:41.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:41.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 secure :-1 s=READY pgs=3791 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f000eef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:41.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 secure :-1 s=READY pgs=3791 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f000eef0).stop 2021-03-09T09:11:41.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:41.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:41.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e2c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:41.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 unknown :-1 s=CLOSED pgs=3791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:41.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:11:41.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:41.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:41.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:41.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:41.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:41.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:41.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:41.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077a40 con 0x7f63e00070a0 2021-03-09T09:11:41.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:41.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019180 con 0x7f6400145bd0 2021-03-09T09:11:41.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:41.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:11:41.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:41.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:41.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:41.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:41.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.161+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:11:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:41.171 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10136 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:41.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:41.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:41.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:41.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:41.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:41.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:41.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:41.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:41.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:11:41.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:11:41.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:41.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:41.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a60 con 0x7f63e005c7c0 2021-03-09T09:11:41.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:41.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:41.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013dc0 con 0x7f63e005c7c0 2021-03-09T09:11:41.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:11:41.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:41.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:11:41.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:41.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:41.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.162+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient: get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:11:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 0 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-03-09T09:11:41.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:41.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:41.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:41.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:11:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:41.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:41.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:41.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:41.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:41.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63e00070a0 2021-03-09T09:11:41.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:41.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00277b0 con 0x7f6400145bd0 2021-03-09T09:11:41.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:41.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014060 con 0x7f6400146050 2021-03-09T09:11:41.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:41.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:41.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:41.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:41.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:11:41.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:41.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:41.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:41.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:41.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:41.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:41.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:41.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:41.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:41.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:41.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:41.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:41.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:11:41.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:11:41.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:41.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:41.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00241c0 con 0x7f6400146050 2021-03-09T09:11:41.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:41.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:41.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:11:41.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:11:41.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:41.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc00b6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:41.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:11:41.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:41.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f6400146050 2021-03-09T09:11:41.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:41.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:41.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:41.381 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:41.381 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:41.381 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:41.382 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:41.382 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:41.382 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:41.382 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:41.382 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:41.383 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:41.383 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:41.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f6400146050 2021-03-09T09:11:41.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:41.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.364+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:11:41.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f6400146050 2021-03-09T09:11:41.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:41.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:41.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:42.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:42.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:42.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:42.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:42.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:42.515+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:43.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:43.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:43.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:43.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:44.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:44.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:44.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:44.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:44.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:44.609+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:45.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:45.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:45.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:45.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:46.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:11:46.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:46.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc00b6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:46.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 secure :-1 s=READY pgs=3818 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc00b6c0).stop 2021-03-09T09:11:46.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:46.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:46.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e2c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:46.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e2c0 unknown :-1 s=CLOSED pgs=3818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:46.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:11:46.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:46.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:46.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:46.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:46.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:46.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:46.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:46.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e980 con 0x7f63e00070a0 2021-03-09T09:11:46.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:46.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e970 con 0x7f6400145bd0 2021-03-09T09:11:46.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:46.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e005c7c0 2021-03-09T09:11:46.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:46.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:46.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:46.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:46.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:46.172 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:46.163+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10175 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:46.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.164+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:46.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.164+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:11:46.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.164+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:46.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.164+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:46.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:46.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:46.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:46.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:46.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:46.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:46.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:46.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:46.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:11:46.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:11:46.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.165+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:46.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:46.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026670 con 0x7f63e005c7c0 2021-03-09T09:11:46.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:46.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:46.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00241c0 con 0x7f63e005c7c0 2021-03-09T09:11:46.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:11:46.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:46.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc064750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:46.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:11:46.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:46.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63e005c7c0 2021-03-09T09:11:46.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:46.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:46.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.166+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:46.181 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:46.181 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:46.182 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:46.182 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:46.182 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:46.182 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:46.183 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:46.183 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:46.183 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:46.183 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:46.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc064360 con 0x7f63e005c7c0 2021-03-09T09:11:46.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.167+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:46.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.167+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:11:46.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.168+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63e005c7c0 2021-03-09T09:11:46.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:46.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:46.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:47.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:47.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:47.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:47.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:48.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:48.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:48.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:48.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:49.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:49.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:49.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:49.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:49.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:49.610+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:50.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:50.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:50.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:50.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:51.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:11:51.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc064750).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=3828 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc064750).stop 2021-03-09T09:11:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:51.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=CLOSED pgs=3828 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:11:51.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:51.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:51.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:51.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:51.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:51.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:51.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:51.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024410 con 0x7f63e00070a0 2021-03-09T09:11:51.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:51.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:11:51.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:51.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018960 con 0x7f6400146050 2021-03-09T09:11:51.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:51.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:51.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.165+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:51.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:51.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:51.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:51.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:11:51.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:51.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:51.176 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:51.166+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10211 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:51.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:51.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:51.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:11:51.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:11:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f6400146050 2021-03-09T09:11:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:11:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:11:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:11:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 rx=0x7f63f000ec50 tx=0x7f63f0030220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.167+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:11:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.169+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.169+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.169+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:51.185 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:51.185 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:51.185 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:51.185 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:51.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006820 con 0x7f6400146050 2021-03-09T09:11:51.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.169+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:51.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.169+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:11:51.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f6400146050 2021-03-09T09:11:51.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:51.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:51.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:52.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:52.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:52.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:52.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:52.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:52.517+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:53.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:53.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:53.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:53.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:54.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:54.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:54.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:54.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:54.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:54.611+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:11:55.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:55.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:55.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:55.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:56.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:11:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:11:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 rx=0x7f63f000ec50 tx=0x7f63f0030220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:11:56.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3834 cs=0 l=1 rev1=1 rx=0x7f63f000ec50 tx=0x7f63f0030220).stop 2021-03-09T09:11:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:11:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:11:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=3834 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:11:56.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:11:56.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:56.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:11:56.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:56.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:11:56.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:11:56.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:11:56.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:56.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00663d0 con 0x7f63e00070a0 2021-03-09T09:11:56.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:56.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010330 con 0x7f6400145bd0 2021-03-09T09:11:56.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:11:56.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00277b0 con 0x7f63e005c7c0 2021-03-09T09:11:56.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:11:56.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:11:56.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:56.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:11:56.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:11:56.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:11:56.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.167+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:11:56.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.168+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:11:56.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.168+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:11:56.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.168+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:11:56.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.168+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:11:56.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.168+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:11:56.177 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:11:56.168+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10250 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:11:56.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.168+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:11:56.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:56.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:56.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:11:56.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:11:56.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:11:56.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:11:56.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:11:56.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:56.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63e005c7c0 2021-03-09T09:11:56.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:11:56.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:11:56.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026670 con 0x7f63e005c7c0 2021-03-09T09:11:56.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:11:56.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:56.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:11:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002a5e0 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc052fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:11:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.169+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:11:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc077180 con 0x7f63e005c7c0 2021-03-09T09:11:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.170+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:11:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.170+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:11:56.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.170+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:11:56.183 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:11:56.183 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:11:56.183 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:56.183 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:11:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:11:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:11:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:11:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:11:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:11:56.185 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:11:56.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc076d30 con 0x7f63e005c7c0 2021-03-09T09:11:56.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.170+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:11:56.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.170+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:11:56.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.171+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc046eb0 con 0x7f63e005c7c0 2021-03-09T09:11:56.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:56.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:56.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:57.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:57.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:57.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:57.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:58.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:58.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:58.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:58.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:59.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:59.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:11:59.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:59.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:11:59.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:11:59.612+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:00.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:00.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:00.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:00.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:01.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:12:01.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:01.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.168+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002a5e0 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc052fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:01.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002a5e0 secure :-1 s=READY pgs=3849 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc052fa0).stop 2021-03-09T09:12:01.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:01.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:01.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002a5e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:01.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002a5e0 unknown :-1 s=CLOSED pgs=3849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:01.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:12:01.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:01.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:01.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:01.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:01.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:01.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:01.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:01.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ab00 con 0x7f63e00070a0 2021-03-09T09:12:01.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:01.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e970 con 0x7f6400145bd0 2021-03-09T09:12:01.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:01.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:12:01.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:01.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:01.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:01.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:01.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.169+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:01.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.170+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:01.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.170+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:12:01.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.170+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:01.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.170+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:01.181 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:01.170+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10282 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:01.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.170+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:01.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.170+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:01.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.170+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:01.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:01.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:01.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:01.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:01.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:01.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:12:01.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:12:01.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:01.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:01.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00112f0 con 0x7f6400146050 2021-03-09T09:12:01.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:01.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:01.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T09:12:01.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:12:01.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.171+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:01.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.172+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 rx=0x7f63e006cf00 tx=0x7f63fc00b5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:01.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.172+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:12:01.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.172+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:01.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc032b50 con 0x7f6400146050 2021-03-09T09:12:01.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.173+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.173+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.173+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:01.190 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:01.190 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:01.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.174+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032b50 con 0x7f6400146050 2021-03-09T09:12:01.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.174+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:01.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.174+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:12:01.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.176+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T09:12:01.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:01.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:01.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:02.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:02.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:02.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:02.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:02.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:02.518+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:03.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:03.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:03.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:03.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:04.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:04.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:04.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:04.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:04.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:04.613+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:05.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:05.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:05.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:05.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:06.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:12:06.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:06.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 rx=0x7f63e006cf00 tx=0x7f63fc00b5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:06.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3868 cs=0 l=1 rev1=1 rx=0x7f63e006cf00 tx=0x7f63fc00b5d0).stop 2021-03-09T09:12:06.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:06.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:06.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:06.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=3868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:06.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:12:06.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:06.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:06.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:06.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:06.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:06.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:06.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:06.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066260 con 0x7f63e00070a0 2021-03-09T09:12:06.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:06.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cad0 con 0x7f6400145bd0 2021-03-09T09:12:06.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:06.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00277b0 con 0x7f63e005c7c0 2021-03-09T09:12:06.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.170+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:06.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.171+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:06.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.171+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:06.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.171+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:06.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.171+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:06.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.171+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:06.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.171+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:12:06.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.171+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:06.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.171+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:06.180 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10305 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:06.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:06.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:06.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:06.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:06.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:06.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:06.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:06.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:06.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:12:06.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:12:06.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:06.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:06.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:12:06.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:06.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:06.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63e005c7c0 2021-03-09T09:12:06.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:12:06.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.172+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:06.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.173+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:06.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.173+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:12:06.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.173+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:06.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.174+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e005c7c0 2021-03-09T09:12:06.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.174+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:06.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.174+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:06.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.174+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:06.188 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:06.188 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:06.188 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:06.189 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:06.189 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:06.189 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:06.189 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:06.189 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:06.190 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:06.190 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:06.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.174+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027e70 con 0x7f63e005c7c0 2021-03-09T09:12:06.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.174+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:06.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.174+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:12:06.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.176+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f63e005c7c0 2021-03-09T09:12:06.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:06.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:06.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:07.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:07.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:07.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:07.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:08.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:08.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:08.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:08.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:09.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:09.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:09.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:09.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:09.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:09.614+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:10.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:10.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:10.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:10.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:11.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:12:11.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:11.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:11.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 secure :-1 s=READY pgs=3874 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000bc0).stop 2021-03-09T09:12:11.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:11.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:11.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:11.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 unknown :-1 s=CLOSED pgs=3874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:11.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:12:11.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:11.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:11.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:11.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:11.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00663d0 con 0x7f63e00070a0 2021-03-09T09:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e470 con 0x7f6400145bd0 2021-03-09T09:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f6400146050 2021-03-09T09:12:11.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.173+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.173+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:11.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.173+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.173+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.173+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.173+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:11.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.173+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:11.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.174+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:11.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.174+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:11.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.174+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:11.183 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:11.174+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10362 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:11.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:11.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:11.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:11.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:11.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:11.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:12:11.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:12:11.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:11.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:11.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d8a0 con 0x7f6400146050 2021-03-09T09:12:11.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:11.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:11.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:12:11.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:12:11.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.175+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:11.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.176+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3897 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f4016ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:11.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.176+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:12:11.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.176+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:11.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:12:11.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.177+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:11.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.177+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:11.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.177+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:11.188 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:11.189 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:11.190 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:11.190 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:11.190 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:11.190 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:11.191 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:11.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f6400146050 2021-03-09T09:12:11.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.179+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:11.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.179+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:12:11.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:12:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:11.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:11.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:12.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:12.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:12.519+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:13.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:13.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:13.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:13.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:14.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:14.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:14.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:14.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:14.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:14.614+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:15.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:15.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:15.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:15.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:16.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:12:16.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:16.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3897 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f4016ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:16.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3897 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f4016ec0).stop 2021-03-09T09:12:16.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:16.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:16.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:16.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=3897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:16.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:12:16.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:16.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:16.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:16.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:16.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:16.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:16.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:16.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:12:16.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:16.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c670 con 0x7f6400145bd0 2021-03-09T09:12:16.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:16.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00277b0 con 0x7f63e005c7c0 2021-03-09T09:12:16.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.173+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:16.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:16.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:16.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:16.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:16.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:16.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:12:16.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:16.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:16.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:16.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:16.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:16.186 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:16.174+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10399 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:16.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:16.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:16.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:16.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:16.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:16.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:12:16.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:12:16.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:16.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:16.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066210 con 0x7f63e005c7c0 2021-03-09T09:12:16.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:16.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:16.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00112f0 con 0x7f63e005c7c0 2021-03-09T09:12:16.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:12:16.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.175+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:16.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:12:16.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:16.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=3902 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc0242b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:16.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e005c7c0 2021-03-09T09:12:16.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:16.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:16.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:16.193 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:16.193 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:16.194 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:16.194 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:16.194 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:16.194 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:16.194 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:16.195 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:16.195 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:16.195 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:16.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e005c7c0 2021-03-09T09:12:16.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:16.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.176+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:12:16.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e005c7c0 2021-03-09T09:12:16.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:16.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:16.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:17.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:17.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:17.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:17.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:18.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:18.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:19.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:19.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:19.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:19.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:19.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:19.615+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:20.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:20.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:20.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:20.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:21.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:12:21.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:21.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=3902 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc0242b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:21.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=3902 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc0242b0).stop 2021-03-09T09:12:21.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:21.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:21.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:21.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=CLOSED pgs=3902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:21.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:12:21.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:21.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:21.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:21.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:21.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:21.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:21.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:21.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b210 con 0x7f63e00070a0 2021-03-09T09:12:21.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:21.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f6400145bd0 2021-03-09T09:12:21.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:21.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bb00 con 0x7f6400146050 2021-03-09T09:12:21.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:21.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:21.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:21.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:21.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.175+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:21.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.176+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:21.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.176+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:12:21.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.176+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:21.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.176+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:21.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.176+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:21.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.176+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:21.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.176+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:21.190 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:21.176+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10438 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:21.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:21.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:21.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:21.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:12:21.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:12:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:21.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0069630 con 0x7f6400146050 2021-03-09T09:12:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:21.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:12:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.178+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:12:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.178+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:21.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.178+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:12:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.178+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.178+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f004c260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001a1e0 con 0x7f6400146050 2021-03-09T09:12:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.179+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.179+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.179+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:21.204 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003b480 con 0x7f6400146050 2021-03-09T09:12:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.181+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.181+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:12:21.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001a1e0 con 0x7f6400146050 2021-03-09T09:12:21.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:21.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:21.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:22.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:22.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:22.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:22.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:22.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:22.519+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:23.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:23.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:24.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:24.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:24.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:24.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:24.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:24.616+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:25.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:25.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:25.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:25.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:26.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:12:26.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:26.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f004c260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:26.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3922 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f004c260).stop 2021-03-09T09:12:26.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:26.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:26.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:26.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=3922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:26.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:12:26.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:26.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:26.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:26.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:26.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:26.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:26.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:26.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00279b0 con 0x7f63e00070a0 2021-03-09T09:12:26.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:26.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:12:26.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:26.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63e005c7c0 2021-03-09T09:12:26.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:26.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:26.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.177+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:26.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:26.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:12:26.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:26.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:26.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:26.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:26.188 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10459 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:26.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:26.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:26.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.178+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:26.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:26.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:26.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:26.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:26.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:26.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:12:26.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:12:26.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:26.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:26.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:12:26.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:26.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:26.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d8a0 con 0x7f63e005c7c0 2021-03-09T09:12:26.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:12:26.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.179+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:26.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.180+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:12:26.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.180+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d740 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f40036c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:26.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.180+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:26.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4036bc0 con 0x7f63e005c7c0 2021-03-09T09:12:26.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.181+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:26.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.181+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:26.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.181+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:26.194 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:26.195 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:26.195 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:26.195 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:26.195 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:26.195 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:26.196 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:26.196 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:26.196 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:26.196 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:26.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4036d20 con 0x7f63e005c7c0 2021-03-09T09:12:26.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.181+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:26.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.181+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:12:26.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4036bc0 con 0x7f63e005c7c0 2021-03-09T09:12:26.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:26.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:26.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:27.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:27.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:27.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:27.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:28.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:28.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:28.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:28.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:29.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:29.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:29.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:29.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:29.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:29.617+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:30.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:30.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:30.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:30.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:31.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:12:31.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:31.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d740 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f40036c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:31.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d740 secure :-1 s=READY pgs=3934 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f40036c0).stop 2021-03-09T09:12:31.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:31.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:31.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:31.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d740 unknown :-1 s=CLOSED pgs=3934 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:31.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:12:31.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:31.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:31.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:31.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:31.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:31.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:31.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:31.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002dc80 con 0x7f63e00070a0 2021-03-09T09:12:31.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:31.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006cb90 con 0x7f6400145bd0 2021-03-09T09:12:31.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:31.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00138f0 con 0x7f6400146050 2021-03-09T09:12:31.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:31.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:31.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:31.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:31.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.179+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:31.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.180+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:31.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.180+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:12:31.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.180+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:31.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.180+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:31.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.180+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:31.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.180+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:31.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.180+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:31.190 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:31.180+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10513 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:31.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:31.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:31.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:31.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:31.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:31.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:12:31.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:12:31.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:31.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:31.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d8a0 con 0x7f6400146050 2021-03-09T09:12:31.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:31.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:31.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:12:31.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:12:31.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.181+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:31.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3946 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f405b400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:31.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:12:31.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:31.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b860 con 0x7f6400146050 2021-03-09T09:12:31.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:31.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:31.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:31.196 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:31.196 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:31.197 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:31.197 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:31.197 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:31.197 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:31.197 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:31.198 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:31.198 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:31.198 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:31.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b860 con 0x7f6400146050 2021-03-09T09:12:31.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:31.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.182+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:12:31.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402a400 con 0x7f6400146050 2021-03-09T09:12:31.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:31.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:31.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:32.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:32.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:32.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:32.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:32.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:32.521+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:33.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:33.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:33.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:33.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:34.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:34.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:34.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:34.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:34.618+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:35.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:35.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:35.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:35.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.180+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:12:36.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.180+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:36.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.180+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3946 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f405b400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:36.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3946 cs=0 l=1 rev1=1 rx=0x7f63f4047280 tx=0x7f63f405b400).stop 2021-03-09T09:12:36.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:36.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:36.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:36.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=3946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:36.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:12:36.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:36.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:36.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:36.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:36.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:36.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:36.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:36.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e00070a0 2021-03-09T09:12:36.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:36.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016240 con 0x7f6400145bd0 2021-03-09T09:12:36.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:36.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078b10 con 0x7f63e005c7c0 2021-03-09T09:12:36.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:36.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:36.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:36.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:36.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.181+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:36.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.182+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:36.195 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:36.182+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10549 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:36.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.182+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:12:36.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.182+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:36.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.182+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:36.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.182+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:36.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.182+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:36.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.182+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:36.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:36.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:36.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:36.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:36.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:36.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:12:36.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:12:36.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:36.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:36.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:12:36.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:36.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:36.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0069630 con 0x7f63e005c7c0 2021-03-09T09:12:36.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:12:36.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.183+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:36.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.184+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ab00 secure :-1 s=READY pgs=3961 cs=0 l=1 rev1=1 rx=0x7f63e002a510 tx=0x7f63f0013f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:36.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.184+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:12:36.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.184+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:36.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002d360 con 0x7f63e005c7c0 2021-03-09T09:12:36.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.185+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:36.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.185+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:36.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.185+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:36.203 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:36.204 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:36.205 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:36.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e005c7c0 2021-03-09T09:12:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.185+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.185+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:12:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.186+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0006780 con 0x7f63e005c7c0 2021-03-09T09:12:36.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:36.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:36.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:37.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:37.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:37.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:37.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:38.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:38.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:38.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:38.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:39.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:39.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:39.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:39.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:39.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:39.619+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:40.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:40.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:40.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:40.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:41.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.182+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:12:41.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.182+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:41.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.182+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ab00 secure :-1 s=READY pgs=3961 cs=0 l=1 rev1=1 rx=0x7f63e002a510 tx=0x7f63f0013f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:41.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.182+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ab00 secure :-1 s=READY pgs=3961 cs=0 l=1 rev1=1 rx=0x7f63e002a510 tx=0x7f63f0013f00).stop 2021-03-09T09:12:41.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:41.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:41.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006ab00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:41.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006ab00 unknown :-1 s=CLOSED pgs=3961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:41.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:12:41.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:41.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:41.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:41.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:41.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:41.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:41.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:41.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:12:41.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:41.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f6400145bd0 2021-03-09T09:12:41.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:41.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400146050 2021-03-09T09:12:41.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:41.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:41.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:41.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:41.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.183+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:41.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:41.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:12:41.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:41.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:41.193 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10588 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:41.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:41.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:41.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:41.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:41.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:41.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:41.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:41.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:41.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:12:41.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:12:41.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:41.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:41.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e0d0 con 0x7f6400146050 2021-03-09T09:12:41.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:41.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:41.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d8a0 con 0x7f6400146050 2021-03-09T09:12:41.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:12:41.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.184+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:41.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.185+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3972 cs=0 l=1 rev1=1 rx=0x7f63f405b7f0 tx=0x7f63f405b780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:41.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.185+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:12:41.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.185+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:41.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f6400146050 2021-03-09T09:12:41.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.187+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:41.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.187+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:41.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.187+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:41.201 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:41.201 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:41.201 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:41.201 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:41.202 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:41.202 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:41.202 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:41.202 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:41.203 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:41.203 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:41.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.189+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f6400146050 2021-03-09T09:12:41.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.189+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:41.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.189+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:12:41.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4069d80 con 0x7f6400146050 2021-03-09T09:12:41.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:41.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:41.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:42.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:42.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:42.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:42.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:42.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:42.522+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:43.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:43.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:43.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:43.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:44.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:44.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:44.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:44.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:44.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:44.620+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:45.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:45.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:45.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:46.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:12:46.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:46.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3972 cs=0 l=1 rev1=1 rx=0x7f63f405b7f0 tx=0x7f63f405b780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:46.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=3972 cs=0 l=1 rev1=1 rx=0x7f63f405b7f0 tx=0x7f63f405b780).stop 2021-03-09T09:12:46.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:46.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:46.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:46.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=3972 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:46.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:12:46.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:46.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:46.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:46.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:46.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:46.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:46.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:46.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002dc80 con 0x7f63e00070a0 2021-03-09T09:12:46.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:46.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ebc0 con 0x7f6400145bd0 2021-03-09T09:12:46.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:46.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00174c0 con 0x7f63e005c7c0 2021-03-09T09:12:46.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.184+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:46.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.185+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:46.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.185+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:46.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.185+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:46.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.185+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:46.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:46.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:12:46.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:46.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:46.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:46.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:46.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:46.204 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10622 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:46.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:46.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:46.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:46.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:46.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:46.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:12:46.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:12:46.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:46.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:46.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63e005c7c0 2021-03-09T09:12:46.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:46.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:46.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:12:46.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:12:46.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.186+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:46.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:12:46.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=3985 cs=0 l=1 rev1=1 rx=0x7f63e0011ea0 tx=0x7f63f0040ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:46.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:46.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002d290 con 0x7f63e005c7c0 2021-03-09T09:12:46.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:46.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:46.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:46.211 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:46.212 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:46.212 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:46.212 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:46.212 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:46.213 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:46.213 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:46.213 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:46.213 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:46.213 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:46.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002d3f0 con 0x7f63e005c7c0 2021-03-09T09:12:46.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:46.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.187+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:12:46.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0054050 con 0x7f63e005c7c0 2021-03-09T09:12:46.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:46.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:46.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:47.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:47.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:47.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:47.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:48.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:48.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:48.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:48.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:49.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:49.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:49.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:49.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:49.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:49.621+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:50.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:50.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:50.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:50.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:51.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:12:51.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:51.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=3985 cs=0 l=1 rev1=1 rx=0x7f63e0011ea0 tx=0x7f63f0040ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:51.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=3985 cs=0 l=1 rev1=1 rx=0x7f63e0011ea0 tx=0x7f63f0040ec0).stop 2021-03-09T09:12:51.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:51.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:51.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:51.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=CLOSED pgs=3985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:51.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:12:51.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:51.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:51.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:51.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:51.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:51.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:51.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:51.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e00070a0 2021-03-09T09:12:51.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:51.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016fd0 con 0x7f6400145bd0 2021-03-09T09:12:51.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:51.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:12:51.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.186+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:51.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:51.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.187+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.187+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.187+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:51.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.187+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:51.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.187+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:12:51.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.187+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:51.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.187+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:51.197 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:51.188+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10663 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.188+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.188+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.188+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:51.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:51.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:12:51.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:12:51.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:51.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:51.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:12:51.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:51.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:51.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066210 con 0x7f6400146050 2021-03-09T09:12:51.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:12:51.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.189+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:51.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.190+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:12:51.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.190+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc046b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:51.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.190+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:51.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:12:51.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.192+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:51.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.192+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.192+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:51.204 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:51.205 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:51.206 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0583e0 con 0x7f6400146050 2021-03-09T09:12:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.194+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.194+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:12:51.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:12:51.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:51.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:51.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:52.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:52.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:52.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:52.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:52.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:52.524+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:53.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:53.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:53.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:53.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:54.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:54.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:54.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:54.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:54.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:54.622+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:12:55.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:55.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:55.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:55.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:56.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:12:56.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:12:56.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc046b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:12:56.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4002 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc046b50).stop 2021-03-09T09:12:56.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:56.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:12:56.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:12:56.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=4002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:12:56.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:12:56.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:56.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:12:56.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:56.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:12:56.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:12:56.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:12:56.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:56.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a8f0 con 0x7f63e00070a0 2021-03-09T09:12:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018e40 con 0x7f6400145bd0 2021-03-09T09:12:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:12:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e005c7c0 2021-03-09T09:12:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:12:56.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:12:56.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:56.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.188+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:12:56.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:12:56.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:12:56.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:12:56.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:12:56.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:12:56.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:12:56.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:12:56.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:12:56.199 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10699 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:12:56.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:12:56.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:56.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:56.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:12:56.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:12:56.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:12:56.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:12:56.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.189+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:12:56.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001cfb0 con 0x7f63e005c7c0 2021-03-09T09:12:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:12:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:12:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63e005c7c0 2021-03-09T09:12:56.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:12:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:12:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 rx=0x7f63e0038f10 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:12:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.190+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:12:56.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003f3b0 con 0x7f63e005c7c0 2021-03-09T09:12:56.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.191+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:12:56.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.191+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:12:56.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.191+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:12:56.205 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:12:56.206 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:12:56.206 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:56.206 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:12:56.206 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:12:56.206 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:12:56.207 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:12:56.207 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:12:56.207 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:12:56.207 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:12:56.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003f1d0 con 0x7f63e005c7c0 2021-03-09T09:12:56.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.191+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:12:56.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.191+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:12:56.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e005c7c0 2021-03-09T09:12:56.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:56.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:56.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:57.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:57.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:57.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:57.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:58.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:58.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:58.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:58.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:59.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:59.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:12:59.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:59.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:12:59.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:12:59.623+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:00.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:00.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:00.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:00.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:01.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:13:01.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:01.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 rx=0x7f63e0038f10 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:01.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=4014 cs=0 l=1 rev1=1 rx=0x7f63e0038f10 tx=0x7f63f0000c30).stop 2021-03-09T09:13:01.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:01.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:01.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:01.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=CLOSED pgs=4014 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:01.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:13:01.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:01.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:01.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:01.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:01.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:01.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:01.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d480 con 0x7f63e00070a0 2021-03-09T09:13:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:13:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f6400146050 2021-03-09T09:13:01.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:01.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:01.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:01.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:01.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.190+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:01.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.191+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:01.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.191+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:13:01.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.191+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:01.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.191+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:01.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.191+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:01.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.191+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:01.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.191+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:01.202 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:01.191+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10733 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:01.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:01.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:01.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:01.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:01.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:01.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:13:01.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:13:01.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:01.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:01.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0008840 con 0x7f6400146050 2021-03-09T09:13:01.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:01.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:01.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001cfb0 con 0x7f6400146050 2021-03-09T09:13:01.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:13:01.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.192+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:01.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f0036d70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:01.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:13:01.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:01.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002d4a0 con 0x7f6400146050 2021-03-09T09:13:01.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:01.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:01.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:01.208 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:01.208 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:01.208 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:01.209 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:01.209 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:01.209 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:01.209 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:01.209 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:01.210 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:01.210 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:01.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000e260 con 0x7f6400146050 2021-03-09T09:13:01.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:01.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.193+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:13:01.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00036b0 con 0x7f6400146050 2021-03-09T09:13:01.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:01.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:01.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:02.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:02.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:02.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:02.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:02.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:02.525+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:03.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:03.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:03.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:03.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:04.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:04.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:04.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:04.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:04.624+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:05.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:05.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:05.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:05.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:06.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:06.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:06.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f0036d70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:06.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4028 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f0036d70).stop 2021-03-09T09:13:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=4028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:06.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:13:06.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:06.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:06.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0011b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:06.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:06.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:06.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:06.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019bb0 con 0x7f63e00070a0 2021-03-09T09:13:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f0a0 con 0x7f6400145bd0 2021-03-09T09:13:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aee0 con 0x7f63e005c7c0 2021-03-09T09:13:06.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:06.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:06.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:06.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:06.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.192+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:06.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.193+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0011b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:06.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.193+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:13:06.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.193+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:06.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.193+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:06.202 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:06.193+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10772 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:06.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.193+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:06.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.193+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:06.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.193+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:06.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.194+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:06.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:06.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:06.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:06.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:06.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:13:06.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:13:06.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:06.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:06.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:13:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:13:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:13:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0011b10 secure :-1 s=READY pgs=4048 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc006db0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:13:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.195+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.197+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:13:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.197+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:06.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.197+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:06.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.197+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:06.209 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:06.209 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:06.210 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:06.210 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:06.210 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:06.210 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:06.210 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:06.211 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:06.211 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:06.211 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:06.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0583e0 con 0x7f63e005c7c0 2021-03-09T09:13:06.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.199+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:06.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.199+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:13:06.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:13:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:06.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:07.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:07.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:08.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:08.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:08.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:08.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:09.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:09.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:09.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:09.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:09.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:09.625+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:10.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:10.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:10.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:10.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:11.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0011b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:11.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0011b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:11.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0011b10 secure :-1 s=READY pgs=4048 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc006db0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:11.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0011b10 secure :-1 s=READY pgs=4048 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc006db0).stop 2021-03-09T09:13:11.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:11.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:11.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0011b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:11.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0011b10 unknown :-1 s=CLOSED pgs=4048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:11.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:13:11.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:11.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:11.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:11.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:11.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:11.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:11.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:11.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b450 con 0x7f63e00070a0 2021-03-09T09:13:11.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:11.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:13:11.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:11.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023790 con 0x7f6400146050 2021-03-09T09:13:11.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:11.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:11.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:11.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:11.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.194+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:11.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:11.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.195+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:13:11.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.195+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.195+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:11.209 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:11.195+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10803 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:11.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.195+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:11.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.195+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.195+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:13:11.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:13:11.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:11.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:13:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:13:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:13:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.196+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.197+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc02be10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:11.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.197+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:13:11.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.197+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:11.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.198+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03e1e0 con 0x7f6400146050 2021-03-09T09:13:11.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.198+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:11.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.198+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:11.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.198+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:11.218 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:11.221 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03e1e0 con 0x7f6400146050 2021-03-09T09:13:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.199+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.199+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:13:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0641a0 con 0x7f6400146050 2021-03-09T09:13:11.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:11.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:11.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:12.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:12.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:12.526+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:13.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:13.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:13.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:13.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:14.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:14.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:14.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:14.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:14.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:14.626+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:15.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:15.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:15.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:15.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:13:16.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:16.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc02be10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:16.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4059 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc02be10).stop 2021-03-09T09:13:16.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:16.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=4059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:13:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.195+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:16.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:16.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:16.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:16.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:16.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:16.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:16.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aee0 con 0x7f63e00070a0 2021-03-09T09:13:16.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:16.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de80 con 0x7f6400145bd0 2021-03-09T09:13:16.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000caa0 con 0x7f63e005c7c0 2021-03-09T09:13:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:16.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:13:16.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:16.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:16.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:16.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:16.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.196+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:16.207 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10849 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:16.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:16.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:16.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:16.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:16.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:16.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:13:16.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:13:16.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:16.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:16.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f5f0 con 0x7f63e005c7c0 2021-03-09T09:13:16.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:16.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:16.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e0d0 con 0x7f63e005c7c0 2021-03-09T09:13:16.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:13:16.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.197+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:16.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.198+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:13:16.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.198+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:16.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.198+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=4068 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f400ff30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:16.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e005c7c0 2021-03-09T09:13:16.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.200+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:16.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.200+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:16.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.200+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:16.213 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:16.213 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:16.213 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:16.213 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:16.214 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:16.214 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:16.214 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:16.214 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:16.214 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:16.215 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:16.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005c10 con 0x7f63e005c7c0 2021-03-09T09:13:16.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.200+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:16.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.200+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:13:16.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.201+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:13:16.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:16.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:16.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:17.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:17.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:17.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:17.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:18.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:18.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:18.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:18.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:19.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:19.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:19.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:19.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:19.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:19.627+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:20.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:20.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:20.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:20.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:21.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=4068 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f400ff30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=4068 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f400ff30).stop 2021-03-09T09:13:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:21.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=CLOSED pgs=4068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:13:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:21.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:21.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e00070a0 2021-03-09T09:13:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:13:21.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:21.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028710 con 0x7f6400146050 2021-03-09T09:13:21.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:21.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:21.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:21.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.197+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:21.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:21.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.198+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:21.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.198+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:13:21.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.198+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:21.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.198+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:21.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.198+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:21.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.198+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:21.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.198+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:21.208 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:21.199+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10881 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:21.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.199+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:21.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.199+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:21.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:21.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:21.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:21.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:13:21.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:13:21.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:21.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:21.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:13:21.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:21.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:21.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:13:21.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:13:21.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:21.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:13:21.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 rx=0x7f63e002ec90 tx=0x7f63fc052f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:21.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.200+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:21.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.201+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc008a40 con 0x7f6400146050 2021-03-09T09:13:21.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.201+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:21.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.201+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:21.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.201+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:21.214 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:21.214 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:21.215 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:21.215 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:21.215 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:21.215 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:21.216 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:21.216 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:21.216 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:21.216 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:21.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02be80 con 0x7f6400146050 2021-03-09T09:13:21.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.202+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:21.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.202+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:13:21.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f6400146050 2021-03-09T09:13:21.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:21.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:21.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:22.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:22.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:22.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:22.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:22.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:22.528+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:23.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:23.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:23.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:23.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:24.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:24.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:24.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:24.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:24.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:24.628+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:25.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:25.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:25.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:25.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:26.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:26.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:26.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 rx=0x7f63e002ec90 tx=0x7f63fc052f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:26.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4084 cs=0 l=1 rev1=1 rx=0x7f63e002ec90 tx=0x7f63fc052f30).stop 2021-03-09T09:13:26.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:26.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:26.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:26.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=4084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:26.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:13:26.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:26.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:26.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:26.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:26.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:26.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:26.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:26.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ffd0 con 0x7f63e00070a0 2021-03-09T09:13:26.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:26.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023790 con 0x7f6400145bd0 2021-03-09T09:13:26.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:26.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f0a0 con 0x7f63e005c7c0 2021-03-09T09:13:26.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:26.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:26.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.199+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:26.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.200+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:26.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.200+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:26.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.200+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:26.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.201+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:13:26.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.201+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:26.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.201+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:26.210 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:26.200+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10922 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:26.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.201+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:26.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.201+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:26.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.201+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:26.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:26.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:26.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:26.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:26.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:26.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:13:26.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:13:26.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:26.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:26.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e110 con 0x7f63e005c7c0 2021-03-09T09:13:26.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:26.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:26.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0008840 con 0x7f63e005c7c0 2021-03-09T09:13:26.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:13:26.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:26.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 rx=0x7f63f005e070 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:26.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.203+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:13:26.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.203+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:26.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003f1d0 con 0x7f63e005c7c0 2021-03-09T09:13:26.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.204+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:26.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.205+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:26.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.205+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:26.217 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:26.217 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:26.217 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:26.217 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:26.217 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:26.218 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:26.218 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:26.218 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:26.218 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:26.218 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:26.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00157f0 con 0x7f63e005c7c0 2021-03-09T09:13:26.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.208+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:26.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.208+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:13:26.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003f1d0 con 0x7f63e005c7c0 2021-03-09T09:13:26.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:26.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:26.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:27.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:27.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:27.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:27.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:28.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:28.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:28.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:28.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:29.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:29.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:29.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:29.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:29.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:29.629+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:30.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:30.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:30.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:30.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:31.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.201+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:13:31.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.201+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:31.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.201+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 rx=0x7f63f005e070 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:31.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.201+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 secure :-1 s=READY pgs=4099 cs=0 l=1 rev1=1 rx=0x7f63f005e070 tx=0x7f63f0013b80).stop 2021-03-09T09:13:31.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:31.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:31.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:31.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d5a0 unknown :-1 s=CLOSED pgs=4099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:31.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:13:31.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:31.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:31.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:31.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:31.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:31.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:31.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:31.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aee0 con 0x7f63e00070a0 2021-03-09T09:13:31.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:31.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010ab0 con 0x7f6400145bd0 2021-03-09T09:13:31.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:31.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018b60 con 0x7f6400146050 2021-03-09T09:13:31.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:31.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:31.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:31.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:31.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:31.212 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:31.202+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10953 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:31.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.203+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:31.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.203+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:13:31.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.203+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:31.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.203+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:31.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.203+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:31.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.203+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:31.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.203+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:31.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:31.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:31.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:31.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:31.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:31.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:13:31.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:13:31.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:31.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:31.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018880 con 0x7f6400146050 2021-03-09T09:13:31.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:31.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:31.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:13:31.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:13:31.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:31.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:13:31.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:31.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.205+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4119 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:31.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc008a40 con 0x7f6400146050 2021-03-09T09:13:31.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.206+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:31.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.206+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:31.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.206+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:31.219 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:31.220 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:31.220 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:31.220 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:31.220 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:31.221 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:31.221 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:31.221 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:31.221 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:31.221 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:31.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc008a40 con 0x7f6400146050 2021-03-09T09:13:31.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.209+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:31.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.209+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:13:31.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f6400146050 2021-03-09T09:13:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:31.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:32.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:32.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:32.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:32.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:32.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:32.529+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:33.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:33.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:33.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:33.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:34.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:34.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:34.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:34.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:34.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:34.630+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:35.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:35.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:35.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:35.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.203+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.203+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:36.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.203+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4119 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.203+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 secure :-1 s=READY pgs=4119 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc003a50).stop 2021-03-09T09:13:36.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069140 unknown :-1 s=CLOSED pgs=4119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:36.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:13:36.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:36.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:36.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:36.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:13:36.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:36.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00123f0 con 0x7f6400145bd0 2021-03-09T09:13:36.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:36.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e005c7c0 2021-03-09T09:13:36.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:36.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:36.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:36.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:36.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:36.213 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:36.204+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 10999 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:36.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.205+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:36.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.205+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:13:36.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.205+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:36.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.205+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:36.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.205+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:36.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.205+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:36.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.205+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:36.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:36.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:36.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:36.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:36.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:36.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:13:36.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:13:36.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:36.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:36.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:13:36.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:36.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:36.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f5f0 con 0x7f63e005c7c0 2021-03-09T09:13:36.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:13:36.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.206+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:36.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.207+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:13:36.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.207+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:36.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.207+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:36.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.212+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4018400 con 0x7f63e005c7c0 2021-03-09T09:13:36.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.212+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:36.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.212+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:36.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.212+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:36.221 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:36.221 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:36.222 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:36.222 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:36.222 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:36.222 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:36.222 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:36.223 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:36.223 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:36.223 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:36.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.212+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4018560 con 0x7f63e005c7c0 2021-03-09T09:13:36.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.212+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:36.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.212+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:13:36.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:13:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:36.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:37.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:37.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:37.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:37.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:38.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:38.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:38.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:38.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:39.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:39.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:39.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:39.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:39.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:39.631+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:40.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:40.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:40.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:40.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:41.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:13:41.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:41.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003bb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:41.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 secure :-1 s=READY pgs=4130 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003bb0).stop 2021-03-09T09:13:41.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:41.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:41.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007d8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:41.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007d8e0 unknown :-1 s=CLOSED pgs=4130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:41.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:13:41.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:41.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:41.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:41.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:41.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:41.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:41.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:41.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008850 con 0x7f63e00070a0 2021-03-09T09:13:41.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:41.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008a90 con 0x7f6400145bd0 2021-03-09T09:13:41.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:41.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010530 con 0x7f6400146050 2021-03-09T09:13:41.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:41.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:41.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:41.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.205+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:41.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.206+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:41.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.206+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:41.215 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:41.206+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11036 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:41.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.206+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:13:41.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.206+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:41.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.206+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:41.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:41.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:41.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:41.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:41.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:41.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:41.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:41.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:41.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:13:41.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:13:41.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:41.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:41.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000edd0 con 0x7f6400146050 2021-03-09T09:13:41.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:41.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:41.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:13:41.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:13:41.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.207+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:41.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.208+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 rx=0x7f63f4061770 tx=0x7f63f40612f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:41.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.208+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:13:41.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.208+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:41.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.210+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072ef0 con 0x7f6400146050 2021-03-09T09:13:41.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.210+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:41.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.210+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:41.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.210+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:41.223 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:41.223 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:41.223 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:41.223 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:41.224 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:41.224 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:41.224 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:41.224 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:41.225 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:41.225 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:41.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072ef0 con 0x7f6400146050 2021-03-09T09:13:41.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.211+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:41.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.212+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:13:41.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:13:41.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:41.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:41.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:42.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:42.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:42.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:42.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:42.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:42.531+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:43.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:43.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:43.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:43.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:44.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:44.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:44.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:44.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:44.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:44.632+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:45.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:45.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:46.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:46.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:46.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 rx=0x7f63f4061770 tx=0x7f63f40612f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:46.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 secure :-1 s=READY pgs=4139 cs=0 l=1 rev1=1 rx=0x7f63f4061770 tx=0x7f63f40612f0).stop 2021-03-09T09:13:46.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:46.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:46.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:46.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 unknown :-1 s=CLOSED pgs=4139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:46.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:13:46.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:46.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:46.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:46.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:46.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:46.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:46.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:46.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e00070a0 2021-03-09T09:13:46.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:46.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:13:46.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:46.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001ab0 con 0x7f63e005c7c0 2021-03-09T09:13:46.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:46.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:46.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:46.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:46.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.207+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:46.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.208+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:46.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.208+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:13:46.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.208+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:46.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.208+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:46.218 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:46.208+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11074 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:46.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.208+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:46.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.208+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:46.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.208+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:46.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:46.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:46.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:46.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:46.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:46.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:13:46.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:13:46.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:46.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:46.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00142a0 con 0x7f63e005c7c0 2021-03-09T09:13:46.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:46.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:46.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e110 con 0x7f63e005c7c0 2021-03-09T09:13:46.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:13:46.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.209+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:46.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.210+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:13:46.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.210+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:46.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.210+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027fc0 secure :-1 s=READY pgs=4153 cs=0 l=1 rev1=1 rx=0x7f63f0036cb0 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:46.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036ce0 con 0x7f63e005c7c0 2021-03-09T09:13:46.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.211+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:46.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.211+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:46.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.211+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:46.226 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:46.226 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:46.227 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:46.227 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:46.227 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:46.228 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:46.228 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:46.228 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:46.228 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:46.228 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:46.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00541f0 con 0x7f63e005c7c0 2021-03-09T09:13:46.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.211+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:46.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.211+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:13:46.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003f1d0 con 0x7f63e005c7c0 2021-03-09T09:13:46.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:46.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:46.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:47.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:47.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:47.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:47.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:48.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:48.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:48.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:48.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:49.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:49.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:49.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:49.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:49.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:49.633+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:50.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:50.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:50.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:50.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0027fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:13:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0027fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027fc0 secure :-1 s=READY pgs=4153 cs=0 l=1 rev1=1 rx=0x7f63f0036cb0 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:51.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027fc0 secure :-1 s=READY pgs=4153 cs=0 l=1 rev1=1 rx=0x7f63f0036cb0 tx=0x7f63f004d180).stop 2021-03-09T09:13:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0027fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:51.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0027fc0 unknown :-1 s=CLOSED pgs=4153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:13:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:51.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:51.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:51.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:51.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:51.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bf50 con 0x7f63e00070a0 2021-03-09T09:13:51.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:51.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bbc0 con 0x7f6400145bd0 2021-03-09T09:13:51.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:51.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010530 con 0x7f6400146050 2021-03-09T09:13:51.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:51.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:51.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:51.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:51.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:51.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.210+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:51.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.210+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:13:51.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.210+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:51.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.210+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:51.220 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:51.210+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11111 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:51.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.210+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:51.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.210+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:51.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.210+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:51.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:51.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:51.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:51.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:51.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:51.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:13:51.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:13:51.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:51.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:51.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T09:13:51.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:51.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:51.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000edd0 con 0x7f6400146050 2021-03-09T09:13:51.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:13:51.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.211+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:51.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.212+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 secure :-1 s=READY pgs=4166 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4023280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:51.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.212+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:13:51.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.212+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:51.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034a30 con 0x7f6400146050 2021-03-09T09:13:51.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.213+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:51.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.213+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:51.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.213+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:51.227 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:51.227 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:51.228 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:51.228 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:51.228 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:51.228 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:51.228 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:51.229 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:51.229 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:51.229 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:51.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034c60 con 0x7f6400146050 2021-03-09T09:13:51.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.213+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.213+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:13:51.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.214+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f6400146050 2021-03-09T09:13:51.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:51.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:51.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:52.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:52.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:52.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:52.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:52.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:52.532+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:53.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:53.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:53.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:53.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:54.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:54.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:54.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:54.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:54.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:54.634+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:13:55.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:55.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:55.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:55.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:56.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:13:56.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:13:56.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 secure :-1 s=READY pgs=4166 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4023280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:13:56.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 secure :-1 s=READY pgs=4166 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4023280).stop 2021-03-09T09:13:56.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:56.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:13:56.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:13:56.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 unknown :-1 s=CLOSED pgs=4166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:13:56.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:13:56.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:56.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:13:56.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:56.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:13:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:13:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:13:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008a90 con 0x7f63e00070a0 2021-03-09T09:13:56.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:56.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:13:56.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:13:56.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078600 con 0x7f63e005c7c0 2021-03-09T09:13:56.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:13:56.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:13:56.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:56.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:13:56.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.211+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:13:56.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:13:56.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:13:56.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:13:56.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:13:56.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:13:56.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:13:56.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:13:56.223 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11144 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:13:56.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:13:56.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:56.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:56.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:13:56.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:13:56.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:13:56.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.212+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:13:56.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:13:56.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:56.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:13:56.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:13:56.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:13:56.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63e005c7c0 2021-03-09T09:13:56.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:13:56.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:56.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:13:56.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:13:56.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.213+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006a6c0 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 rx=0x7f63f402a7c0 tx=0x7f63f4036340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:13:56.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.214+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402a400 con 0x7f63e005c7c0 2021-03-09T09:13:56.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.214+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:13:56.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.214+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:13:56.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.214+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:13:56.229 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:13:56.229 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:13:56.230 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:56.230 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:13:56.230 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:13:56.230 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:13:56.230 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:13:56.231 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:13:56.231 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:13:56.231 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:13:56.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f402a400 con 0x7f63e005c7c0 2021-03-09T09:13:56.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.215+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:13:56.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.215+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:13:56.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f63e005c7c0 2021-03-09T09:13:56.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:56.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:56.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:57.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:57.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:57.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:57.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:58.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:58.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:58.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:58.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:59.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:59.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:13:59.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:59.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:13:59.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:13:59.635+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:00.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:00.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:00.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:00.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:01.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:14:01.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:01.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006a6c0 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 rx=0x7f63f402a7c0 tx=0x7f63f4036340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:01.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006a6c0 secure :-1 s=READY pgs=4178 cs=0 l=1 rev1=1 rx=0x7f63f402a7c0 tx=0x7f63f4036340).stop 2021-03-09T09:14:01.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:01.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:01.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e006a6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:01.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e006a6c0 unknown :-1 s=CLOSED pgs=4178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:01.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:14:01.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:01.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:01.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:01.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:01.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:01.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:01.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:01.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001ab0 con 0x7f63e00070a0 2021-03-09T09:14:01.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:01.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400145bd0 2021-03-09T09:14:01.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:01.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065b70 con 0x7f6400146050 2021-03-09T09:14:01.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:14:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:01.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.213+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:01.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:01.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:01.223 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11179 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:01.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:01.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:01.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:01.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:01.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:01.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:01.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:01.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:14:01.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:14:01.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:01.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:01.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d180 con 0x7f6400146050 2021-03-09T09:14:01.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:01.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:01.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018880 con 0x7f6400146050 2021-03-09T09:14:01.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:14:01.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.214+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:01.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.215+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc0522d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:01.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.215+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:14:01.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.215+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:01.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:14:01.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.216+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:01.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.216+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:01.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.216+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:01.230 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:01.230 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:01.231 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:01.231 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:01.231 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:01.231 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:01.231 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:01.232 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:01.232 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:01.232 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:01.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0711d0 con 0x7f6400146050 2021-03-09T09:14:01.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.216+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:01.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.216+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:14:01.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.219+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:14:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:01.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:01.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:02.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:02.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:02.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:02.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:02.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:02.533+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:03.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:03.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:03.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:03.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:04.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:04.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:04.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:04.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:04.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:04.636+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:05.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:05.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:05.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:05.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:06.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:14:06.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:06.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc0522d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:06.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 secure :-1 s=READY pgs=4191 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc0522d0).stop 2021-03-09T09:14:06.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:06.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:06.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:06.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001bd50 unknown :-1 s=CLOSED pgs=4191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:06.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:14:06.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:06.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:06.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:06.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:06.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:06.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:06.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:06.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bbc0 con 0x7f63e00070a0 2021-03-09T09:14:06.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:06.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00101d0 con 0x7f6400145bd0 2021-03-09T09:14:06.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:06.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e005c7c0 2021-03-09T09:14:06.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:06.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:06.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:06.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:06.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:06.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.215+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:06.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:14:06.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:06.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:06.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:06.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:06.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:06.225 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11219 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:06.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:06.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:06.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:06.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:06.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:06.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:14:06.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:14:06.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:06.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:06.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017310 con 0x7f63e005c7c0 2021-03-09T09:14:06.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:06.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:06.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00142a0 con 0x7f63e005c7c0 2021-03-09T09:14:06.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:14:06.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.216+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:06.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.217+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c240 secure :-1 s=READY pgs=4204 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f002d3e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:06.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.217+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:14:06.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.217+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:06.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.219+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0003b10 con 0x7f63e005c7c0 2021-03-09T09:14:06.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.219+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:06.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.219+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:06.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.219+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:06.231 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:06.232 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:06.232 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:06.232 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:06.232 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:06.233 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:06.233 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:06.233 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:06.233 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:06.233 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:06.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0019140 con 0x7f63e005c7c0 2021-03-09T09:14:06.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.220+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:06.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.220+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:14:06.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.221+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0003b10 con 0x7f63e005c7c0 2021-03-09T09:14:06.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:06.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:06.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:07.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:07.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:07.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:07.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:08.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:08.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:08.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:08.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:09.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:09.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:09.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:09.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:09.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:09.637+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:10.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:10.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:10.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:10.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:14:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c240 secure :-1 s=READY pgs=4204 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f002d3e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c240 secure :-1 s=READY pgs=4204 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f002d3e0).stop 2021-03-09T09:14:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001c240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001c240 unknown :-1 s=CLOSED pgs=4204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:14:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:11.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:11.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:14:11.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:11.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065b70 con 0x7f6400145bd0 2021-03-09T09:14:11.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:11.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001d50 con 0x7f6400146050 2021-03-09T09:14:11.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:11.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:11.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:11.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:11.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:14:11.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:11.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:11.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:11.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:11.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.217+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:11.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:11.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:11.228 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11258 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:11.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:11.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:11.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:11.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:11.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:11.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:14:11.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:14:11.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:11.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:11.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:14:11.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:11.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:11.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:14:11.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:14:11.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.218+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:11.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.219+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:14:11.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.219+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 secure :-1 s=READY pgs=4220 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f402a400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:11.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.219+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:11.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:14:11.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.222+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:11.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.222+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:11.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.222+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:11.234 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:11.234 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:11.234 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:11.234 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:11.235 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:11.235 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:11.235 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:11.235 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:11.235 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:11.236 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:11.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f6400146050 2021-03-09T09:14:11.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.222+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:11.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.222+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:14:11.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f6400146050 2021-03-09T09:14:11.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:11.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:11.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:12.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:12.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:12.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:12.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:12.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:12.535+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:13.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:13.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:13.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:13.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:14.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:14.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:14.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:14.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:14.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:14.638+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:15.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:15.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:15.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:15.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:16.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:14:16.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 secure :-1 s=READY pgs=4220 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f402a400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 secure :-1 s=READY pgs=4220 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f402a400).stop 2021-03-09T09:14:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000dfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:16.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 unknown :-1 s=CLOSED pgs=4220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:16.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:14:16.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:16.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:16.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:16.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:16.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:16.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:16.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:16.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e2b0 con 0x7f63e00070a0 2021-03-09T09:14:16.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:16.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f6400145bd0 2021-03-09T09:14:16.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:16.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bbc0 con 0x7f63e005c7c0 2021-03-09T09:14:16.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:16.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:16.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:16.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:16.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:16.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:14:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:16.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:16.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.219+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:16.229 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11296 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:16.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:16.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:16.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:16.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:16.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:16.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:14:16.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:14:16.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:16.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:16.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:14:16.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:16.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:16.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d180 con 0x7f63e005c7c0 2021-03-09T09:14:16.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:14:16.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.220+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:16.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.221+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 secure :-1 s=READY pgs=4233 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:16.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.221+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:14:16.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.221+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:16.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071220 con 0x7f63e005c7c0 2021-03-09T09:14:16.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.223+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:16.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.223+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:16.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.223+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:16.236 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:16.236 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:16.236 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:16.236 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:16.237 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:16.237 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:16.237 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:16.237 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:16.237 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:16.238 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:16.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071220 con 0x7f63e005c7c0 2021-03-09T09:14:16.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.223+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:16.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.223+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:14:16.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.225+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02c220 con 0x7f63e005c7c0 2021-03-09T09:14:16.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:16.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:16.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:17.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:17.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:17.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:17.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:18.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:18.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:18.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:18.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:19.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:19.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:19.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:19.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:19.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:19.639+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:20.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:20.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:20.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:20.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:21.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:14:21.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:21.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 secure :-1 s=READY pgs=4233 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:21.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 secure :-1 s=READY pgs=4233 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc04a740).stop 2021-03-09T09:14:21.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:21.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:21.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:21.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 unknown :-1 s=CLOSED pgs=4233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:21.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:14:21.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:21.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:21.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:21.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:21.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:21.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:21.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:21.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f290 con 0x7f63e00070a0 2021-03-09T09:14:21.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:21.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078600 con 0x7f6400145bd0 2021-03-09T09:14:21.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:21.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:14:21.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:21.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:21.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:21.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:21.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:21.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:21.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:14:21.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:21.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:21.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:21.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:21.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:21.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:21.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:21.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:21.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:21.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:21.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:14:21.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:14:21.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:21.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:21.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:14:21.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:21.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:21.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:14:21.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:14:21.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:21.235 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:21.221+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11334 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:21.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.222+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 secure :-1 s=READY pgs=4244 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4062f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:21.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.222+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:14:21.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.222+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:21.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404be90 con 0x7f6400146050 2021-03-09T09:14:21.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.223+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:21.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.223+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:21.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.223+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:21.237 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:21.238 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:21.238 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:21.238 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:21.238 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:21.239 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:21.239 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:21.239 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:21.239 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:21.239 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:21.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046e50 con 0x7f6400146050 2021-03-09T09:14:21.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.223+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:21.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.223+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:14:21.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f6400146050 2021-03-09T09:14:21.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:21.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:21.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:22.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:22.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:22.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:22.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:22.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:22.536+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:23.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:23.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:23.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:23.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:24.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:24.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:24.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:24.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:24.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:24.640+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:25.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:25.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:25.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:25.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:26.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:14:26.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:26.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 secure :-1 s=READY pgs=4244 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4062f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:26.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 secure :-1 s=READY pgs=4244 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4062f70).stop 2021-03-09T09:14:26.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:26.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:26.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000dfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:26.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000dfb0 unknown :-1 s=CLOSED pgs=4244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:26.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:14:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0014440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:26.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:26.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:26.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aee0 con 0x7f63e00070a0 2021-03-09T09:14:26.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:26.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008930 con 0x7f6400145bd0 2021-03-09T09:14:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bbc0 con 0x7f63e005c7c0 2021-03-09T09:14:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.222+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:26.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.223+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0014440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:26.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.223+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:14:26.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.223+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:26.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.223+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:26.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.223+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:26.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.223+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:26.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.223+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:26.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.223+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:26.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.223+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:26.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:26.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:14:26.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:14:26.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:26.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:26.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029f60 con 0x7f63e005c7c0 2021-03-09T09:14:26.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:26.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:26.237 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11374 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:26.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:14:26.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:14:26.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.224+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:26.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.225+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0014440 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f40226d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:26.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.225+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:14:26.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.225+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:26.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.227+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4069480 con 0x7f63e005c7c0 2021-03-09T09:14:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.227+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.227+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.227+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:26.241 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:26.241 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:26.241 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:26.241 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:26.242 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:26.242 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:26.242 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:26.242 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:26.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.230+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022b30 con 0x7f63e005c7c0 2021-03-09T09:14:26.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.230+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:26.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.230+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:14:26.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400eb40 con 0x7f63e005c7c0 2021-03-09T09:14:26.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:26.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:26.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:27.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:27.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:27.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:27.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:28.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:28.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:28.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:28.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:29.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:29.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:29.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:29.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:29.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:29.641+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:30.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:30.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:30.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:30.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:31.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0014440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:14:31.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0014440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:31.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0014440 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f40226d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:31.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0014440 secure :-1 s=READY pgs=4260 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f40226d0).stop 2021-03-09T09:14:31.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:31.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:31.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0014440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:31.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0014440 unknown :-1 s=CLOSED pgs=4260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:31.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:14:31.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:31.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:31.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:31.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:31.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:31.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:31.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:31.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e2b0 con 0x7f63e00070a0 2021-03-09T09:14:31.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:31.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f6400145bd0 2021-03-09T09:14:31.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:31.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:14:31.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:31.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:31.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:31.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:31.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:14:31.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:31.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:31.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:31.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:31.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:31.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:31.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:31.235 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:31.225+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11411 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:31.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:31.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:31.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:31.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:31.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:14:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:14:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:14:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:14:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:14:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.226+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:31.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.227+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:14:31.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.227+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ed10 secure :-1 s=READY pgs=4274 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:31.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.227+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:31.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:14:31.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.229+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:31.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.229+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:31.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.229+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:31.241 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:31.241 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:31.242 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:31.242 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:31.242 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:31.242 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:31.242 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:31.243 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:31.243 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:31.243 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:31.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f6400146050 2021-03-09T09:14:31.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.229+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:31.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.229+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:14:31.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0233c0 con 0x7f6400146050 2021-03-09T09:14:31.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:31.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:31.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:32.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:32.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:32.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:32.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:32.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:32.537+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:33.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:33.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:33.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:33.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:34.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:34.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:34.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:34.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:34.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:34.642+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:35.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:35.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:35.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:35.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:36.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:14:36.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:36.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ed10 secure :-1 s=READY pgs=4274 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:36.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ed10 secure :-1 s=READY pgs=4274 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009990).stop 2021-03-09T09:14:36.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:36.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:36.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000ed10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:36.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ed10 unknown :-1 s=CLOSED pgs=4274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:36.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:14:36.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:36.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:36.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:36.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:36.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:36.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:36.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011db0 con 0x7f63e00070a0 2021-03-09T09:14:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aee0 con 0x7f6400145bd0 2021-03-09T09:14:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:36.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027000 con 0x7f63e005c7c0 2021-03-09T09:14:36.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:36.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:36.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:36.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:36.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:36.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.226+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:36.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.227+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:14:36.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.227+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:36.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.227+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:36.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.227+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:36.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.227+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:36.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.227+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:36.237 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:36.227+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11449 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:36.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:36.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:36.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:36.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:36.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:36.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:14:36.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:14:36.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:36.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:36.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:14:36.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:36.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:36.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f820 con 0x7f63e005c7c0 2021-03-09T09:14:36.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:14:36.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.228+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:36.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.231+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002c3e0 secure :-1 s=READY pgs=4292 cs=0 l=1 rev1=1 rx=0x7f63fc052360 tx=0x7f63fc02c4f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:36.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.231+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:14:36.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.231+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e005c7c0 2021-03-09T09:14:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.234+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.234+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.234+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:36.244 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:36.244 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:36.244 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:36.244 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:36.245 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:36.245 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:36.245 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:36.245 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:36.245 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:36.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0475e0 con 0x7f63e005c7c0 2021-03-09T09:14:36.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.234+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:36.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0529c0 con 0x7f63e005c7c0 2021-03-09T09:14:36.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.234+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:14:36.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:36.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:36.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:37.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:37.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:37.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:37.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:38.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:38.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:38.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:38.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:39.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:39.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:39.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:39.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:39.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:39.643+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:40.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:40.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:40.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:40.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:41.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:14:41.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:41.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002c3e0 secure :-1 s=READY pgs=4292 cs=0 l=1 rev1=1 rx=0x7f63fc052360 tx=0x7f63fc02c4f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:41.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002c3e0 secure :-1 s=READY pgs=4292 cs=0 l=1 rev1=1 rx=0x7f63fc052360 tx=0x7f63fc02c4f0).stop 2021-03-09T09:14:41.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:41.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:41.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002c3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:41.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002c3e0 unknown :-1 s=CLOSED pgs=4292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:41.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:14:41.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:41.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:41.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:41.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:41.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:41.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:41.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:41.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ddb0 con 0x7f63e00070a0 2021-03-09T09:14:41.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:41.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067330 con 0x7f6400145bd0 2021-03-09T09:14:41.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:41.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b710 con 0x7f6400146050 2021-03-09T09:14:41.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:41.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:41.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:41.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:41.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.229+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:41.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:41.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:14:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:41.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:41.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:41.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:41.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:41.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:41.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:14:41.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:14:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:41.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d520 con 0x7f6400146050 2021-03-09T09:14:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:41.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:41.243 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11484 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:41.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:14:41.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:14:41.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.230+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:41.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 secure :-1 s=READY pgs=4300 cs=0 l=1 rev1=1 rx=0x7f63fc053010 tx=0x7f63fc032fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:41.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:14:41.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:41.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0327d0 con 0x7f6400146050 2021-03-09T09:14:41.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:41.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:41.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:41.247 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:41.247 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:41.247 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:41.247 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:41.247 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:41.248 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f6400146050 2021-03-09T09:14:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.231+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:14:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f6400146050 2021-03-09T09:14:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:41.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:42.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:42.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:42.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:42.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:42.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:42.539+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:43.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:43.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:43.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:43.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:44.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:44.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:44.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:44.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:44.644+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:45.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:45.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:45.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:45.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:46.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:14:46.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:46.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 secure :-1 s=READY pgs=4300 cs=0 l=1 rev1=1 rx=0x7f63fc053010 tx=0x7f63fc032fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:46.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 secure :-1 s=READY pgs=4300 cs=0 l=1 rev1=1 rx=0x7f63fc053010 tx=0x7f63fc032fa0).stop 2021-03-09T09:14:46.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:46.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:46.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:46.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067d90 unknown :-1 s=CLOSED pgs=4300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:46.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:14:46.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:46.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:46.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:46.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:46.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:46.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:46.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:46.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bbc0 con 0x7f63e00070a0 2021-03-09T09:14:46.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:46.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:14:46.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:46.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ddb0 con 0x7f63e005c7c0 2021-03-09T09:14:46.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:46.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:46.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:46.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:46.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:46.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:46.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:14:46.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:46.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:46.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:46.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:46.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.231+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:46.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:46.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:46.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:46.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:46.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:46.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:14:46.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:14:46.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:46.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:46.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b940 con 0x7f63e005c7c0 2021-03-09T09:14:46.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:46.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:46.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017310 con 0x7f63e005c7c0 2021-03-09T09:14:46.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:14:46.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:46.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068280 secure :-1 s=READY pgs=4313 cs=0 l=1 rev1=1 rx=0x7f63e0010870 tx=0x7f63f0006bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:46.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:14:46.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:46.246 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:46.232+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11521 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:46.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003f400 con 0x7f63e005c7c0 2021-03-09T09:14:46.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.233+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:46.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.233+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:46.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.233+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:46.247 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:46.248 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:46.248 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:46.248 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:46.248 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:46.248 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:46.249 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:46.249 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:46.249 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:46.249 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:46.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0003b10 con 0x7f63e005c7c0 2021-03-09T09:14:46.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.233+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:46.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.233+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:14:46.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.236+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003f2a0 con 0x7f63e005c7c0 2021-03-09T09:14:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:46.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:47.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:47.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:47.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:47.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:48.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:48.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:48.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:48.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:49.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:49.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:49.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:49.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:49.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:49.645+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:50.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:50.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:50.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:50.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0068280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:14:51.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0068280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:51.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068280 secure :-1 s=READY pgs=4313 cs=0 l=1 rev1=1 rx=0x7f63e0010870 tx=0x7f63f0006bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:51.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068280 secure :-1 s=READY pgs=4313 cs=0 l=1 rev1=1 rx=0x7f63e0010870 tx=0x7f63f0006bc0).stop 2021-03-09T09:14:51.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:51.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:51.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0068280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:51.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068280 unknown :-1 s=CLOSED pgs=4313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:51.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:14:51.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:51.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:51.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:51.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065d70 con 0x7f63e00070a0 2021-03-09T09:14:51.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:51.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077540 con 0x7f6400145bd0 2021-03-09T09:14:51.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:51.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:14:51.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:51.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:51.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:51.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:51.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:51.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:51.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:14:51.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:51.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.233+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:51.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:51.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:51.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:51.243 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11561 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:51.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:51.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:51.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:51.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:51.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:51.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:14:51.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:14:51.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:51.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:51.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00112d0 con 0x7f6400146050 2021-03-09T09:14:51.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:51.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:51.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d520 con 0x7f6400146050 2021-03-09T09:14:51.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:14:51.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.234+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:51.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.235+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:14:51.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.235+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017120 secure :-1 s=READY pgs=4326 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc052260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:51.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.235+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:51.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.236+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04f7d0 con 0x7f6400146050 2021-03-09T09:14:51.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.236+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:51.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.236+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:51.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.236+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:51.249 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:51.249 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:51.249 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:51.250 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:51.250 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:51.250 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:51.250 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:51.251 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:51.251 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:51.251 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:51.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.236+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:14:51.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.236+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:51.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.236+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:14:51.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f6400146050 2021-03-09T09:14:51.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:51.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:51.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:52.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:52.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:52.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:52.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:52.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:52.540+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:53.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:53.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:53.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:53.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:54.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:54.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:54.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:54.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:54.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:54.646+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:14:55.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:55.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:55.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:55.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:56.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:14:56.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:14:56.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017120 secure :-1 s=READY pgs=4326 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc052260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:14:56.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017120 secure :-1 s=READY pgs=4326 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc052260).stop 2021-03-09T09:14:56.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:56.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:14:56.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:14:56.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017120 unknown :-1 s=CLOSED pgs=4326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:14:56.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:14:56.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:56.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:14:56.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:56.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:14:56.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:14:56.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:14:56.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:56.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c8d0 con 0x7f63e00070a0 2021-03-09T09:14:56.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:56.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011db0 con 0x7f6400145bd0 2021-03-09T09:14:56.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:14:56.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ddb0 con 0x7f63e005c7c0 2021-03-09T09:14:56.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.234+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:14:56.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.235+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:14:56.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.235+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:56.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.235+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:14:56.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.235+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:14:56.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:14:56.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:14:56.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:14:56.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:14:56.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:14:56.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:14:56.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:14:56.247 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11594 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:14:56.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:14:56.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:56.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:56.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:14:56.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:14:56.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:14:56.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:14:56.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:14:56.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:56.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:14:56.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:14:56.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:14:56.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00112d0 con 0x7f63e005c7c0 2021-03-09T09:14:56.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:14:56.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.236+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:56.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.237+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:14:56.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.237+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067d40 secure :-1 s=READY pgs=4342 cs=0 l=1 rev1=1 rx=0x7f63e0030610 tx=0x7f63fc05df20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:14:56.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.237+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:14:56.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0664a0 con 0x7f63e005c7c0 2021-03-09T09:14:56.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.239+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:14:56.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.239+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:14:56.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.239+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:14:56.254 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:14:56.255 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:14:56.255 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:56.255 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:14:56.255 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:14:56.256 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:14:56.256 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:14:56.256 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:14:56.256 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:14:56.256 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:14:56.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f63e005c7c0 2021-03-09T09:14:56.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.239+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:14:56.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.239+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:14:56.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.241+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012b80 con 0x7f63e005c7c0 2021-03-09T09:14:56.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:56.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:56.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:57.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:57.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:57.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:57.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:58.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:58.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:58.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:58.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:59.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:59.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:14:59.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:59.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:14:59.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:14:59.647+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:00.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:00.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:00.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:00.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:01.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:01.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067d40 secure :-1 s=READY pgs=4342 cs=0 l=1 rev1=1 rx=0x7f63e0030610 tx=0x7f63fc05df20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067d40 secure :-1 s=READY pgs=4342 cs=0 l=1 rev1=1 rx=0x7f63e0030610 tx=0x7f63fc05df20).stop 2021-03-09T09:15:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:01.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067d40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067d40 unknown :-1 s=CLOSED pgs=4342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:15:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.236+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:01.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:01.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:01.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:01.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:01.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:01.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:01.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:15:01.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:01.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018120 con 0x7f6400145bd0 2021-03-09T09:15:01.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:01.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022630 con 0x7f6400146050 2021-03-09T09:15:01.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:01.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:01.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:01.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:01.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.237+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:01.245 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:01.238+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11634 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:01.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.238+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:01.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.238+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:15:01.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.238+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:01.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.238+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:01.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.238+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:01.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.238+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:01.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.238+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:01.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:01.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:01.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:01.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:01.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:01.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:15:01.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:15:01.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:01.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:01.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065b70 con 0x7f6400146050 2021-03-09T09:15:01.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:01.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:01.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b940 con 0x7f6400146050 2021-03-09T09:15:01.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:15:01.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.239+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:01.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.240+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:15:01.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.240+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:01.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.240+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068230 secure :-1 s=READY pgs=4359 cs=0 l=1 rev1=1 rx=0x7f63f002f280 tx=0x7f63f001a510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:01.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:15:01.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.244+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:01.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.244+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:01.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.244+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:01.252 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:01.253 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:01.253 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:01.253 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:01.253 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:01.253 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:01.254 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:01.254 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:01.254 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:01.254 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:01.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0015740 con 0x7f6400146050 2021-03-09T09:15:01.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.244+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:01.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.244+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:15:01.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f000e970 con 0x7f6400146050 2021-03-09T09:15:01.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:01.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:01.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:02.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:02.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:02.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:02.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:02.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:02.541+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:03.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:03.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:03.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:03.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:04.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:04.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:04.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:04.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:04.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:04.647+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:05.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:05.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:05.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:05.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:06.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:06.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:06.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068230 secure :-1 s=READY pgs=4359 cs=0 l=1 rev1=1 rx=0x7f63f002f280 tx=0x7f63f001a510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:06.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068230 secure :-1 s=READY pgs=4359 cs=0 l=1 rev1=1 rx=0x7f63f002f280 tx=0x7f63f001a510).stop 2021-03-09T09:15:06.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:06.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:06.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:06.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068230 unknown :-1 s=CLOSED pgs=4359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:06.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:15:06.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:06.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:06.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:06.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:06.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:06.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:06.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:06.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ddb0 con 0x7f63e00070a0 2021-03-09T09:15:06.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:06.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018320 con 0x7f6400145bd0 2021-03-09T09:15:06.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:06.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:15:06.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.237+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:06.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:06.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:06.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:06.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:06.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:06.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:15:06.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:06.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:06.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:06.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:06.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.238+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:06.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:06.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:06.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:06.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:06.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:06.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:15:06.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:15:06.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:06.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:06.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011140 con 0x7f63e005c7c0 2021-03-09T09:15:06.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:06.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:06.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:15:06.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:15:06.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:06.252 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:06.239+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11672 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:06.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 secure :-1 s=READY pgs=4367 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc052340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:06.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:15:06.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:06.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc029eb0 con 0x7f63e005c7c0 2021-03-09T09:15:06.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:06.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:06.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:06.254 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:06.254 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:06.255 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:06.255 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:06.255 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:06.255 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:06.256 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:06.256 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:06.256 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:06.256 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:06.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc029eb0 con 0x7f63e005c7c0 2021-03-09T09:15:06.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:06.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.240+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:15:06.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f63e005c7c0 2021-03-09T09:15:06.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:06.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:06.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:07.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:07.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:07.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:07.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:08.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:08.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:08.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:08.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:09.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:09.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:09.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:09.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:09.648+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:10.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:10.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:10.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:10.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:11.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:11.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:11.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 secure :-1 s=READY pgs=4367 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc052340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:11.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 secure :-1 s=READY pgs=4367 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc052340).stop 2021-03-09T09:15:11.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:11.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:11.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:11.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 unknown :-1 s=CLOSED pgs=4367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:11.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:15:11.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:11.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:11.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:11.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:11.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:11.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:11.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:11.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005eca0 con 0x7f63e00070a0 2021-03-09T09:15:11.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:11.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028630 con 0x7f6400145bd0 2021-03-09T09:15:11.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:11.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025480 con 0x7f6400146050 2021-03-09T09:15:11.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:11.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.239+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:11.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.240+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:11.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.240+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:11.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.240+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.240+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.240+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:15:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.240+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:11.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.240+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:11.250 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:11.241+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11711 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:11.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.241+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:11.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.241+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:11.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.241+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:11.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:11.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:11.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:11.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:11.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:11.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:15:11.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:15:11.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:11.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:11.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022a30 con 0x7f6400146050 2021-03-09T09:15:11.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:11.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:11.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029f60 con 0x7f6400146050 2021-03-09T09:15:11.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:15:11.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.242+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:11.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.243+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067c50 secure :-1 s=READY pgs=4390 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4062e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:11.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.243+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:15:11.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.243+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:11.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:15:11.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.244+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:11.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.244+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:11.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.244+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:11.257 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:11.257 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:11.257 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:11.257 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:11.257 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:11.258 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:11.258 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:11.258 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:11.258 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:11.259 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:11.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:15:11.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.245+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:11.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.245+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:15:11.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4069270 con 0x7f6400146050 2021-03-09T09:15:11.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:11.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:11.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:12.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:12.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:12.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:12.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:12.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:12.542+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:13.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:13.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:13.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:13.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:14.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:14.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:14.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:14.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:14.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:14.649+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:15.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:15.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:15.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:15.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:16.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:16.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:16.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067c50 secure :-1 s=READY pgs=4390 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4062e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:16.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067c50 secure :-1 s=READY pgs=4390 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4062e80).stop 2021-03-09T09:15:16.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:16.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:16.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:16.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067c50 unknown :-1 s=CLOSED pgs=4390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:16.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:15:16.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:16.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:16.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:16.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:16.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:16.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:16.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:16.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:15:16.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:16.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005eca0 con 0x7f6400145bd0 2021-03-09T09:15:16.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:16.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006af00 con 0x7f63e005c7c0 2021-03-09T09:15:16.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:16.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.241+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:16.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:16.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:16.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:16.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:16.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:15:16.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:16.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:16.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:16.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:16.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:16.252 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:16.242+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11742 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:16.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:16.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:16.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:16.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:16.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:16.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:15:16.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:15:16.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:16.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:16.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001ab0 con 0x7f63e005c7c0 2021-03-09T09:15:16.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:16.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:16.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022a30 con 0x7f63e005c7c0 2021-03-09T09:15:16.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:15:16.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:16.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068140 secure :-1 s=READY pgs=4396 cs=0 l=1 rev1=1 rx=0x7f63f404a010 tx=0x7f63f40486c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:16.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.243+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:15:16.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.244+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:16.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048a50 con 0x7f63e005c7c0 2021-03-09T09:15:16.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.245+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:16.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.245+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:16.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.245+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:16.259 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:16.259 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:16.259 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:16.260 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:16.260 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:16.260 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:16.261 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:16.261 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:16.261 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:16.261 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:16.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63e005c7c0 2021-03-09T09:15:16.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.245+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:16.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.245+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:15:16.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.246+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4018400 con 0x7f63e005c7c0 2021-03-09T09:15:16.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:16.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:16.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:17.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:17.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:17.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:17.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:18.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:18.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:18.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:18.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:19.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:19.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:19.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:19.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:19.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:19.650+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:20.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:20.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:20.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:20.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:21.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0068140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:21.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0068140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:21.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068140 secure :-1 s=READY pgs=4396 cs=0 l=1 rev1=1 rx=0x7f63f404a010 tx=0x7f63f40486c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:21.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068140 secure :-1 s=READY pgs=4396 cs=0 l=1 rev1=1 rx=0x7f63f404a010 tx=0x7f63f40486c0).stop 2021-03-09T09:15:21.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:21.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:21.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0068140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:21.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068140 unknown :-1 s=CLOSED pgs=4396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:21.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:15:21.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:21.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:21.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:21.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:21.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:21.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:21.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:21.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022630 con 0x7f63e00070a0 2021-03-09T09:15:21.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:21.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ddb0 con 0x7f6400145bd0 2021-03-09T09:15:21.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:21.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:15:21.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:21.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:21.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:21.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:21.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:21.253 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:21.244+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11786 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:21.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.245+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:21.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.245+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:15:21.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.245+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:21.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.245+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:21.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:21.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:21.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:21.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:21.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:21.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:21.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:21.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:21.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:15:21.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:15:21.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:21.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:21.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0008cd0 con 0x7f6400146050 2021-03-09T09:15:21.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:21.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:21.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011140 con 0x7f6400146050 2021-03-09T09:15:21.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.247+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:15:21.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.247+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:21.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.247+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029f10 secure :-1 s=READY pgs=4419 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:21.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.247+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:15:21.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.247+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:21.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f6400146050 2021-03-09T09:15:21.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.250+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:21.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.250+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:21.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.250+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:21.263 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:21.263 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:21.263 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:21.263 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:21.264 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:21.264 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:21.264 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:21.264 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:21.264 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:21.265 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:21.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0715c0 con 0x7f6400146050 2021-03-09T09:15:21.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.250+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:21.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.250+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:15:21.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f6400146050 2021-03-09T09:15:21.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:21.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:21.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:22.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:22.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:22.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:22.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:22.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:22.543+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:23.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:23.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:23.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:23.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:24.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:24.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:24.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:24.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:24.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:24.651+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:25.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:25.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:25.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:25.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:26.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0029f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:26.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0029f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:26.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029f10 secure :-1 s=READY pgs=4419 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:26.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029f10 secure :-1 s=READY pgs=4419 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc009990).stop 2021-03-09T09:15:26.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:26.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:26.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0029f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:26.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0029f10 unknown :-1 s=CLOSED pgs=4419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:26.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:15:26.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:26.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:26.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:26.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:26.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:26.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:26.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:26.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d740 con 0x7f63e00070a0 2021-03-09T09:15:26.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:26.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400145bd0 2021-03-09T09:15:26.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:26.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:15:26.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:26.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:26.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:26.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:26.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.245+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:26.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.246+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:26.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.246+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:15:26.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.246+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:26.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.246+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:26.255 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:26.246+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11814 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:26.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.246+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:26.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.246+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:26.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.246+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:26.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.247+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:26.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.247+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:26.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.247+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:26.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.247+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:26.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.247+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:26.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.247+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:15:26.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.247+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:15:26.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:26.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:26.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d520 con 0x7f63e005c7c0 2021-03-09T09:15:26.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:26.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:26.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001ab0 con 0x7f63e005c7c0 2021-03-09T09:15:26.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:15:26.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:26.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 secure :-1 s=READY pgs=4429 cs=0 l=1 rev1=1 rx=0x7f63f4017fe0 tx=0x7f63f40035e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:26.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:15:26.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.248+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:26.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4046bc0 con 0x7f63e005c7c0 2021-03-09T09:15:26.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.249+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:26.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.249+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:26.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.249+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:26.263 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:26.263 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:26.263 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:26.263 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:26.263 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:26.264 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:26.264 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:26.264 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:26.264 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:26.265 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:26.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046bc0 con 0x7f63e005c7c0 2021-03-09T09:15:26.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.251+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:26.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.251+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:15:26.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f401a780 con 0x7f63e005c7c0 2021-03-09T09:15:26.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:26.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:26.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:27.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:27.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:27.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:27.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:28.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:28.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:28.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:28.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:29.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:29.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:29.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:29.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:29.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:29.652+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:30.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:30.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:30.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:30.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:31.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:31.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:31.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 secure :-1 s=READY pgs=4429 cs=0 l=1 rev1=1 rx=0x7f63f4017fe0 tx=0x7f63f40035e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:31.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 secure :-1 s=READY pgs=4429 cs=0 l=1 rev1=1 rx=0x7f63f4017fe0 tx=0x7f63f40035e0).stop 2021-03-09T09:15:31.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:31.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:31.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:31.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 unknown :-1 s=CLOSED pgs=4429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:31.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:15:31.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:31.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:31.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:31.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:31.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:31.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:31.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:31.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023790 con 0x7f63e00070a0 2021-03-09T09:15:31.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:31.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017500 con 0x7f6400145bd0 2021-03-09T09:15:31.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:31.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:15:31.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:31.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:31.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.247+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:31.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.248+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:31.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.248+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:31.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.248+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:31.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.248+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:15:31.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.248+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:31.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.248+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:31.257 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:31.248+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11861 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:31.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.249+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:31.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.249+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:31.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.249+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:31.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:31.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:31.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:31.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:31.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:31.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:15:31.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:15:31.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:31.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:31.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:15:31.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:31.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:31.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0008cd0 con 0x7f6400146050 2021-03-09T09:15:31.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:15:31.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.250+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:31.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.251+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:15:31.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.251+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:31.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.251+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 secure :-1 s=READY pgs=4447 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc0323b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:31.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023e70 con 0x7f6400146050 2021-03-09T09:15:31.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.254+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:31.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.254+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:31.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.254+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:31.265 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:31.266 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:31.266 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:31.266 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:31.266 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:31.267 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:31.267 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:31.267 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:31.267 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:31.267 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:31.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.255+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023e70 con 0x7f6400146050 2021-03-09T09:15:31.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.255+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:31.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.255+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:15:31.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.256+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f6400146050 2021-03-09T09:15:31.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:31.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:31.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:32.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:32.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:32.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:32.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:32.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:32.545+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:33.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:33.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:33.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:34.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:34.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:34.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:34.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:34.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:34.653+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:35.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:35.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:35.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:35.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:36.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.249+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:36.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.249+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:36.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.249+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 secure :-1 s=READY pgs=4447 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc0323b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:36.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.249+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 secure :-1 s=READY pgs=4447 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc0323b0).stop 2021-03-09T09:15:36.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.249+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:36.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.249+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:36.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:36.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.249+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 unknown :-1 s=CLOSED pgs=4447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:36.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.249+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:15:36.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:36.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:36.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013e30 con 0x7f63e00070a0 2021-03-09T09:15:36.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:36.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400145bd0 2021-03-09T09:15:36.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:36.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:15:36.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:36.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:36.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:36.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:36.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:36.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:36.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:15:36.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:36.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:36.260 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:36.250+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11898 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:36.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:36.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:36.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:36.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:36.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:36.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:36.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:36.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:36.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:15:36.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:15:36.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:36.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:36.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f820 con 0x7f63e005c7c0 2021-03-09T09:15:36.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:36.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:36.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d520 con 0x7f63e005c7c0 2021-03-09T09:15:36.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:15:36.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.251+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:36.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.252+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:15:36.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.252+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:36.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.252+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 secure :-1 s=READY pgs=4452 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4034b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:36.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.255+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034db0 con 0x7f63e005c7c0 2021-03-09T09:15:36.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.255+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:36.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.255+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:36.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.255+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:36.267 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:36.267 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:36.267 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:36.268 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:36.268 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:36.268 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:36.268 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:36.268 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:36.269 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:36.269 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:36.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.256+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034db0 con 0x7f63e005c7c0 2021-03-09T09:15:36.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.256+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:36.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.256+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:15:36.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:15:36.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:36.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:36.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:37.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:37.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:37.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:37.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:38.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:38.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:38.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:38.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:39.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:39.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:39.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:39.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:39.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:39.654+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:40.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:40.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:40.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:40.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:41.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:15:41.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:41.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 secure :-1 s=READY pgs=4452 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4034b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:41.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 secure :-1 s=READY pgs=4452 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4034b80).stop 2021-03-09T09:15:41.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:41.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:41.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001bd50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:41.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001bd50 unknown :-1 s=CLOSED pgs=4452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:15:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:41.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:41.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:41.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:41.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067610 con 0x7f63e00070a0 2021-03-09T09:15:41.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:41.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023790 con 0x7f6400145bd0 2021-03-09T09:15:41.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:41.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a120 con 0x7f6400146050 2021-03-09T09:15:41.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:41.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:41.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:41.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.251+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:41.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.252+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:41.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.252+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:41.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.252+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:15:41.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.252+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:41.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.252+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:41.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.252+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:41.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.252+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:41.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.252+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:41.262 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:41.252+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11936 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:41.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:41.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:41.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:41.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:41.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:41.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:15:41.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:15:41.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:41.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:41.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001ab0 con 0x7f6400146050 2021-03-09T09:15:41.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:41.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:41.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f820 con 0x7f6400146050 2021-03-09T09:15:41.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:15:41.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:41.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005d2b0 secure :-1 s=READY pgs=4471 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f402ad80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:41.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:15:41.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.254+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:41.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.255+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009350 con 0x7f6400146050 2021-03-09T09:15:41.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.255+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:41.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.255+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:41.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.255+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:41.268 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:41.269 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:41.269 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:41.269 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:41.269 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:41.270 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:41.270 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:41.270 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:41.270 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:41.270 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:41.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4006a90 con 0x7f6400146050 2021-03-09T09:15:41.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.257+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:41.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.257+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:15:41.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.258+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:15:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:41.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:42.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:42.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:42.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:42.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:42.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:42.546+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:43.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:43.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:43.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:43.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:44.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:44.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:44.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:44.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:44.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:44.655+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:45.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:45.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:45.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:45.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:46.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:15:46.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:46.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005d2b0 secure :-1 s=READY pgs=4471 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f402ad80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:46.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005d2b0 secure :-1 s=READY pgs=4471 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f402ad80).stop 2021-03-09T09:15:46.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:46.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:46.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005d2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:46.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005d2b0 unknown :-1 s=CLOSED pgs=4471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:46.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:15:46.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:46.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:46.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:15:46.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:46.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00793e0 con 0x7f6400145bd0 2021-03-09T09:15:46.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:46.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e005c7c0 2021-03-09T09:15:46.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.253+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:46.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:46.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.254+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:15:46.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.254+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:46.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.254+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:46.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.254+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:46.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.254+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:46.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.254+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:46.264 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:46.254+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 11974 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:46.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:46.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:46.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:15:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:15:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:46.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:46.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019c00 con 0x7f63e005c7c0 2021-03-09T09:15:46.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:46.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:46.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065b70 con 0x7f63e005c7c0 2021-03-09T09:15:46.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:15:46.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.255+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:46.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.256+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:15:46.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.256+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:46.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.256+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 secure :-1 s=READY pgs=4480 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f001a250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:46.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00481f0 con 0x7f63e005c7c0 2021-03-09T09:15:46.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.259+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:46.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.259+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:46.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.259+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:46.271 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:46.271 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:46.271 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:46.271 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:46.273 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:46.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e005c7c0 2021-03-09T09:15:46.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.259+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:46.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.260+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:15:46.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.260+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e005c7c0 2021-03-09T09:15:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:46.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:47.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:47.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:47.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:47.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:48.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:48.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:48.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:48.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:49.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:49.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:49.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:49.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:49.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:49.656+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:50.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:50.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:50.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:50.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:51.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:51.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:51.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 secure :-1 s=READY pgs=4480 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f001a250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:51.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 secure :-1 s=READY pgs=4480 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f001a250).stop 2021-03-09T09:15:51.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:51.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:51.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:51.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067c50 unknown :-1 s=CLOSED pgs=4480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:51.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:15:51.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:51.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:51.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:51.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:51.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:51.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:51.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:51.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e00070a0 2021-03-09T09:15:51.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:51.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019420 con 0x7f6400145bd0 2021-03-09T09:15:51.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:51.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:15:51.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.255+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:51.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:51.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:51.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:51.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:51.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:51.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:15:51.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:51.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:51.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:51.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:51.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:51.266 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:51.256+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12009 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:51.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.257+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:51.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:51.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:51.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:51.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:51.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:15:51.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:15:51.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:51.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:51.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0008cd0 con 0x7f6400146050 2021-03-09T09:15:51.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:51.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:51.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019c00 con 0x7f6400146050 2021-03-09T09:15:51.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:15:51.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:51.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:15:51.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:51.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.258+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 secure :-1 s=READY pgs=4497 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63f00406b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:51.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:15:51.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.259+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:51.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.259+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:51.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.259+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:51.273 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:51.273 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:51.273 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:51.273 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:51.274 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:51.274 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:51.274 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:51.274 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:51.274 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:51.275 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:51.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:15:51.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.261+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:51.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.261+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:15:51.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f6400146050 2021-03-09T09:15:51.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:51.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:51.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:52.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:52.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:52.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:52.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:52.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:52.547+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:53.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:53.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:53.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:53.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:54.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:54.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:54.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:54.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:54.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:54.657+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:15:55.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:55.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:55.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:55.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:56.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:15:56.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:15:56.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 secure :-1 s=READY pgs=4497 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63f00406b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:15:56.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 secure :-1 s=READY pgs=4497 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63f00406b0).stop 2021-03-09T09:15:56.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:56.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:15:56.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:15:56.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068140 unknown :-1 s=CLOSED pgs=4497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:15:56.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:15:56.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:56.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:15:56.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0077940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:56.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:15:56.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:15:56.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:15:56.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:56.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011db0 con 0x7f63e00070a0 2021-03-09T09:15:56.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:56.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00793e0 con 0x7f6400145bd0 2021-03-09T09:15:56.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:15:56.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017500 con 0x7f63e005c7c0 2021-03-09T09:15:56.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:15:56.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:15:56.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:56.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:15:56.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.257+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:15:56.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0077940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:15:56.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:15:56.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:15:56.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:15:56.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:15:56.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:15:56.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:15:56.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:15:56.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:56.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:56.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:15:56.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:15:56.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:15:56.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:15:56.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:15:56.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:56.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d60 con 0x7f63e005c7c0 2021-03-09T09:15:56.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:15:56.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:15:56.271 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12039 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:15:56.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:15:56.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:15:56.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.258+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:56.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.259+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0077940 secure :-1 s=READY pgs=4506 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:15:56.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.259+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:15:56.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.259+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:15:56.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e005c7c0 2021-03-09T09:15:56.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.259+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:15:56.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.259+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:15:56.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.259+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:15:56.274 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:15:56.274 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:15:56.275 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:56.275 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:15:56.275 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:15:56.275 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:15:56.275 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:15:56.276 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:15:56.276 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:15:56.276 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:15:56.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0715c0 con 0x7f63e005c7c0 2021-03-09T09:15:56.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.260+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:15:56.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.260+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:15:56.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:15:56.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:56.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:56.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:57.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:57.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:57.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:57.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:58.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:58.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:58.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:58.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:59.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:59.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:15:59.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:59.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:15:59.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:15:59.658+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:00.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:00.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:00.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:00.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:01.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0077940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:16:01.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0077940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:01.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0077940 secure :-1 s=READY pgs=4506 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:01.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0077940 secure :-1 s=READY pgs=4506 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc003a50).stop 2021-03-09T09:16:01.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:01.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:01.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0077940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:01.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0077940 unknown :-1 s=CLOSED pgs=4506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:01.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:16:01.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:01.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:01.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:01.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:01.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:01.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:01.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:01.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077300 con 0x7f63e00070a0 2021-03-09T09:16:01.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:01.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000db40 con 0x7f6400145bd0 2021-03-09T09:16:01.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:01.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:16:01.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:01.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:01.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:01.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:01.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.259+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:01.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.260+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:01.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.260+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:16:01.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.260+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:01.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.260+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:01.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.260+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:01.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.260+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:01.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.260+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:01.270 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:01.260+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12081 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:01.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:01.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:01.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:01.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:01.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:01.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:16:01.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:16:01.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:01.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:01.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:16:01.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:01.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:01.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001ab0 con 0x7f6400146050 2021-03-09T09:16:01.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:16:01.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.261+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:01.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.262+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00786e0 secure :-1 s=READY pgs=4524 cs=0 l=1 rev1=1 rx=0x7f63f402ae50 tx=0x7f63f402aae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:01.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.262+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:16:01.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.262+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:01.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.264+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f6400146050 2021-03-09T09:16:01.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.264+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:01.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.264+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:01.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.264+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:01.276 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:01.276 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:01.277 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:01.277 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:01.277 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:01.277 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:01.277 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:01.278 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:01.278 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:01.278 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:01.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.266+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f6400146050 2021-03-09T09:16:01.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.266+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:01.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.266+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:16:01.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:16:01.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:01.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:01.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:02.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:02.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:02.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:02.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:02.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:02.548+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:03.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:03.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:03.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:03.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:04.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:04.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:04.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:04.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:04.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:04.659+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:05.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:05.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:05.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:05.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:06.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:16:06.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:06.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00786e0 secure :-1 s=READY pgs=4524 cs=0 l=1 rev1=1 rx=0x7f63f402ae50 tx=0x7f63f402aae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:06.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00786e0 secure :-1 s=READY pgs=4524 cs=0 l=1 rev1=1 rx=0x7f63f402ae50 tx=0x7f63f402aae0).stop 2021-03-09T09:16:06.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:06.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:06.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00786e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:06.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00786e0 unknown :-1 s=CLOSED pgs=4524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:06.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:16:06.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:06.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:06.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:06.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:06.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:06.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:06.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:06.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011db0 con 0x7f63e00070a0 2021-03-09T09:16:06.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:06.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cb60 con 0x7f6400145bd0 2021-03-09T09:16:06.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:06.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:16:06.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:06.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:06.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:06.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:06.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.261+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:06.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.262+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:06.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.262+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:16:06.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.262+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:06.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.262+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:06.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.262+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:06.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.262+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:06.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:06.272 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12122 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:06.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:06.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:06.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:06.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:06.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:06.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:16:06.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:16:06.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:06.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:06.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f360 con 0x7f63e005c7c0 2021-03-09T09:16:06.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:06.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:06.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:16:06.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:16:06.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:06.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:16:06.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:06.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.263+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 secure :-1 s=READY pgs=4535 cs=0 l=1 rev1=1 rx=0x7f63f4023a90 tx=0x7f63f4023b70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:06.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402ae50 con 0x7f63e005c7c0 2021-03-09T09:16:06.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.267+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:06.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.267+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:06.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.267+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:06.279 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:06.279 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:06.279 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:06.279 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:06.280 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:06.280 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:06.280 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:06.280 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:06.280 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:06.281 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:06.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e005c7c0 2021-03-09T09:16:06.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.267+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:06.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.267+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:16:06.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:16:06.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:06.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:06.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:07.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:07.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:07.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:07.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:08.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:08.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:08.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:08.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:09.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:09.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:09.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:09.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:09.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:09.660+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:10.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:10.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:10.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:10.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:11.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:16:11.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:11.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 secure :-1 s=READY pgs=4535 cs=0 l=1 rev1=1 rx=0x7f63f4023a90 tx=0x7f63f4023b70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:11.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 secure :-1 s=READY pgs=4535 cs=0 l=1 rev1=1 rx=0x7f63f4023a90 tx=0x7f63f4023b70).stop 2021-03-09T09:16:11.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:11.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:11.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001d740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:11.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 unknown :-1 s=CLOSED pgs=4535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:11.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:16:11.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:11.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:11.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:11.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:11.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:11.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:11.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:11.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c450 con 0x7f63e00070a0 2021-03-09T09:16:11.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:11.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016a50 con 0x7f6400145bd0 2021-03-09T09:16:11.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:11.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400146050 2021-03-09T09:16:11.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:11.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:11.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.263+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:11.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:11.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:11.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:11.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:16:11.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:11.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:11.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:11.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:11.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:11.274 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:11.264+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12159 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:11.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:11.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:11.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:11.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:11.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:11.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:16:11.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:16:11.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:11.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:11.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f0f0 con 0x7f6400146050 2021-03-09T09:16:11.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:11.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:11.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002d60 con 0x7f6400146050 2021-03-09T09:16:11.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:16:11.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.265+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:11.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.266+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:16:11.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.266+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067ad0 secure :-1 s=READY pgs=4548 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc046bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:11.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.266+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:11.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03f450 con 0x7f6400146050 2021-03-09T09:16:11.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.267+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:11.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.267+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:11.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.267+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:11.280 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:11.280 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:11.281 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:11.281 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:11.281 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:11.281 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:11.281 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:11.282 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:11.282 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:11.282 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:11.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.268+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f6400146050 2021-03-09T09:16:11.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.268+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:11.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.268+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:16:11.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:16:11.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:11.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:11.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:12.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:12.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:12.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:12.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:12.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:12.550+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:13.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:13.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:13.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:13.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:14.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:14.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:14.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:14.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:14.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:14.661+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:15.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:15.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:15.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:15.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:16.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:16:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067ad0 secure :-1 s=READY pgs=4548 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc046bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:16.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067ad0 secure :-1 s=READY pgs=4548 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc046bc0).stop 2021-03-09T09:16:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0067ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:16.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0067ad0 unknown :-1 s=CLOSED pgs=4548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:16.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:16:16.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:16.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:16.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:16.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:16.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:16.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:16.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:16.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63e00070a0 2021-03-09T09:16:16.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:16.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014030 con 0x7f6400145bd0 2021-03-09T09:16:16.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:16.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011fb0 con 0x7f63e005c7c0 2021-03-09T09:16:16.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:16.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:16.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:16.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:16.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:16.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.265+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:16.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.266+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:16:16.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.266+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:16.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.266+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:16.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.266+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:16.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.266+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:16.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.266+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:16.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.266+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:16.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.266+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:16.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.266+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:16.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:16.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:16.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:16:16.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:16:16.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:16.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:16.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b6c0 con 0x7f63e005c7c0 2021-03-09T09:16:16.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:16.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:16.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0008cd0 con 0x7f63e005c7c0 2021-03-09T09:16:16.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:16:16.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:16.283 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:16.267+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12198 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:16.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.268+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:16:16.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.268+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:16.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.268+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 secure :-1 s=READY pgs=4564 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:16.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e005c7c0 2021-03-09T09:16:16.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.269+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:16.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.269+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:16.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.269+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:16.286 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:16.286 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:16.286 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:16.286 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:16.287 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:16.287 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:16.287 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:16.287 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:16.288 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:16.288 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:16.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.272+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e005c7c0 2021-03-09T09:16:16.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.272+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:16.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.272+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:16:16.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e005c7c0 2021-03-09T09:16:16.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:16.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:16.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:17.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:17.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:17.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:17.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:18.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:18.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:18.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:18.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:19.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:19.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:19.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:19.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:19.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:19.662+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:20.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:20.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:20.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:21.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:16:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 secure :-1 s=READY pgs=4564 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 secure :-1 s=READY pgs=4564 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f000df50).stop 2021-03-09T09:16:21.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001d740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001d740 unknown :-1 s=CLOSED pgs=4564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:21.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:16:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:21.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e8d0 con 0x7f63e00070a0 2021-03-09T09:16:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:21.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:16:21.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:21.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c890 con 0x7f6400146050 2021-03-09T09:16:21.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:21.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:21.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.267+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:21.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:21.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:21.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:16:21.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:21.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:21.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:21.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:21.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:21.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:21.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:21.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:16:21.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:16:21.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:21.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:21.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001ab0 con 0x7f6400146050 2021-03-09T09:16:21.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:21.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:21.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f0f0 con 0x7f6400146050 2021-03-09T09:16:21.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:16:21.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:21.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 secure :-1 s=READY pgs=4577 cs=0 l=1 rev1=1 rx=0x7f63e0010af0 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:21.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:16:21.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.269+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:21.283 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:21.268+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12236 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:21.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c8a0 con 0x7f6400146050 2021-03-09T09:16:21.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.271+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:21.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.271+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:21.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.271+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:21.285 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:21.285 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:21.285 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:21.286 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:21.287 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f6400146050 2021-03-09T09:16:21.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.271+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.271+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:16:21.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f6400146050 2021-03-09T09:16:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:21.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:22.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:22.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:22.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:22.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:22.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:22.551+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:23.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:23.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:23.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:23.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:24.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:24.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:24.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:24.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:24.663+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:25.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:25.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:25.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:25.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:26.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:16:26.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:26.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 secure :-1 s=READY pgs=4577 cs=0 l=1 rev1=1 rx=0x7f63e0010af0 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:26.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 secure :-1 s=READY pgs=4577 cs=0 l=1 rev1=1 rx=0x7f63e0010af0 tx=0x7f63fc04a6d0).stop 2021-03-09T09:16:26.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:26.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:26.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000ce30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:26.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 unknown :-1 s=CLOSED pgs=4577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:26.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:16:26.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:26.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:26.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:26.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:26.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:26.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:26.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:26.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011db0 con 0x7f63e00070a0 2021-03-09T09:16:26.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:26.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f6400145bd0 2021-03-09T09:16:26.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:26.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e005c7c0 2021-03-09T09:16:26.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:26.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:26.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:26.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:26.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.269+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:26.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.270+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:26.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.270+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:16:26.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.270+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:26.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.270+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:26.279 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:26.270+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12274 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:26.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:26.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:26.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:26.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:26.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:26.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:26.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:26.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:26.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:16:26.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:16:26.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:26.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:26.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029410 con 0x7f63e005c7c0 2021-03-09T09:16:26.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:26.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:26.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b6c0 con 0x7f63e005c7c0 2021-03-09T09:16:26.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:16:26.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.271+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:26.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.272+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:16:26.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.272+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:26.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.272+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067ad0 secure :-1 s=READY pgs=4596 cs=0 l=1 rev1=1 rx=0x7f63e005d180 tx=0x7f63f0013bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:26.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63e005c7c0 2021-03-09T09:16:26.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.273+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:26.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.273+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:26.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.273+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:26.285 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:26.286 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:26.286 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:26.286 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:26.287 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:26.287 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:26.287 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:26.287 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:26.288 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:26.288 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:26.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006170 con 0x7f63e005c7c0 2021-03-09T09:16:26.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.274+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:26.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.274+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:16:26.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63e005c7c0 2021-03-09T09:16:26.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:26.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:26.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:27.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:27.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:27.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:27.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:28.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:28.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:28.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:28.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:29.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:29.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:29.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:29.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:29.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:29.664+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:30.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:30.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:30.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:30.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:31.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:16:31.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:31.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067ad0 secure :-1 s=READY pgs=4596 cs=0 l=1 rev1=1 rx=0x7f63e005d180 tx=0x7f63f0013bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:31.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067ad0 secure :-1 s=READY pgs=4596 cs=0 l=1 rev1=1 rx=0x7f63e005d180 tx=0x7f63f0013bf0).stop 2021-03-09T09:16:31.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:31.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:31.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0067ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:31.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0067ad0 unknown :-1 s=CLOSED pgs=4596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:31.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:16:31.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:31.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:31.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:31.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:31.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:31.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:31.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:31.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:16:31.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:31.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bf70 con 0x7f6400145bd0 2021-03-09T09:16:31.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:31.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c890 con 0x7f6400146050 2021-03-09T09:16:31.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:31.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:31.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:31.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:31.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.271+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:31.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.272+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:31.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.272+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:16:31.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.272+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:31.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.272+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:31.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.272+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:31.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.272+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:31.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.272+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:31.283 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:31.272+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12310 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:31.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:31.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:31.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:31.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:31.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:31.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:16:31.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:16:31.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:31.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:31.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025c60 con 0x7f6400146050 2021-03-09T09:16:31.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:31.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:31.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f360 con 0x7f6400146050 2021-03-09T09:16:31.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:16:31.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.273+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:31.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.275+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 secure :-1 s=READY pgs=4610 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f40632c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:31.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.275+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:16:31.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.275+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:31.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f6400146050 2021-03-09T09:16:31.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.279+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:31.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.279+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:31.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.279+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:31.290 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:31.290 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:31.290 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:31.290 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:31.290 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:31.291 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:31.291 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:31.291 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:31.291 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:31.292 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:31.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f6400146050 2021-03-09T09:16:31.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.279+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:31.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.279+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:16:31.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:16:31.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:31.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:31.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:32.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:32.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:32.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:32.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:32.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:32.553+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:33.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:33.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:33.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:33.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:34.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:34.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:34.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:34.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:34.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:34.665+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:35.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:35.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:35.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:35.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:36.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.273+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:16:36.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.273+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:36.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.273+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 secure :-1 s=READY pgs=4610 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f40632c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:36.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.273+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 secure :-1 s=READY pgs=4610 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f40632c0).stop 2021-03-09T09:16:36.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.273+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:36.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:36.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:36.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b010 unknown :-1 s=CLOSED pgs=4610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:36.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:16:36.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:36.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:36.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:36.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:36.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:36.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:36.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:36.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c450 con 0x7f63e00070a0 2021-03-09T09:16:36.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:36.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023d10 con 0x7f6400145bd0 2021-03-09T09:16:36.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:36.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022a80 con 0x7f63e005c7c0 2021-03-09T09:16:36.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:36.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:36.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:36.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:36.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:36.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:36.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:16:36.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:36.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:36.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:36.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:36.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:36.293 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:36.274+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12347 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:36.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:36.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:36.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:36.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:36.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:36.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:16:36.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:16:36.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:36.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:36.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e005c7c0 2021-03-09T09:16:36.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:36.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:36.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029410 con 0x7f63e005c7c0 2021-03-09T09:16:36.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:16:36.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.275+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:36.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.276+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001e8e0 secure :-1 s=READY pgs=4618 cs=0 l=1 rev1=1 rx=0x7f63e0066a00 tx=0x7f63f00066f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:36.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.276+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:16:36.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.276+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:36.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004ce80 con 0x7f63e005c7c0 2021-03-09T09:16:36.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.277+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:36.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.277+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:36.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.277+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:36.307 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:36.307 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:36.308 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:36.308 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:36.309 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:36.309 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:36.310 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:36.310 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:36.310 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:36.311 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:36.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.278+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004ce80 con 0x7f63e005c7c0 2021-03-09T09:16:36.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.278+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:36.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.278+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:16:36.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.280+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63e005c7c0 2021-03-09T09:16:36.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:36.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:36.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:37.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:37.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:37.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:37.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:38.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:38.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:38.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:38.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:39.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:39.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:39.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:39.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:39.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:39.666+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:40.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:40.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:40.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:40.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:41.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:16:41.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:41.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001e8e0 secure :-1 s=READY pgs=4618 cs=0 l=1 rev1=1 rx=0x7f63e0066a00 tx=0x7f63f00066f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:41.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001e8e0 secure :-1 s=READY pgs=4618 cs=0 l=1 rev1=1 rx=0x7f63e0066a00 tx=0x7f63f00066f0).stop 2021-03-09T09:16:41.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:41.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:41.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e001e8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:41.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e001e8e0 unknown :-1 s=CLOSED pgs=4618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:41.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:16:41.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:41.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:41.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:41.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:41.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:41.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:41.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:41.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:16:41.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:41.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c450 con 0x7f6400145bd0 2021-03-09T09:16:41.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:41.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016a50 con 0x7f6400146050 2021-03-09T09:16:41.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:41.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:41.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:41.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:41.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.275+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:41.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:41.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:16:41.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:41.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:41.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:41.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:41.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:41.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:41.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:41.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:16:41.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:16:41.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.276+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:41.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:41.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f7c0 con 0x7f6400146050 2021-03-09T09:16:41.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:41.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:41.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:16:41.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:16:41.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:41.296 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12388 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:41.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 secure :-1 s=READY pgs=4631 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f001ab60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:41.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:16:41.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.277+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:41.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.278+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f6400146050 2021-03-09T09:16:41.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.278+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:41.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.278+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:41.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.278+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:41.298 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:41.299 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:41.299 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:41.299 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:41.299 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:41.300 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:41.300 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:41.300 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:41.300 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:41.300 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:41.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.278+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004dcc0 con 0x7f6400146050 2021-03-09T09:16:41.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.278+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:41.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.278+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:16:41.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:16:41.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:41.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:41.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:42.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:42.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:42.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:42.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:42.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:42.554+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:43.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:43.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:43.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:43.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:44.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:44.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:44.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:44.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:44.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:44.667+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:45.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:45.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:45.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:45.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:46.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:16:46.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:46.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 secure :-1 s=READY pgs=4631 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f001ab60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:46.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 secure :-1 s=READY pgs=4631 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f001ab60).stop 2021-03-09T09:16:46.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:46.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:46.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000ce30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:46.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000ce30 unknown :-1 s=CLOSED pgs=4631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:46.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:16:46.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:46.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:46.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:46.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:46.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:46.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:46.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:46.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069720 con 0x7f63e00070a0 2021-03-09T09:16:46.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:46.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400145bd0 2021-03-09T09:16:46.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:46.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005eea0 con 0x7f63e005c7c0 2021-03-09T09:16:46.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:46.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:46.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.277+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:46.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.278+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:46.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.278+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:46.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.278+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:46.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.278+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:16:46.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.278+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:46.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.278+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:46.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:46.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:46.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:46.383 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12421 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:46.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:46.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:46.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:46.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:46.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:46.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:16:46.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:16:46.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:46.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:46.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029410 con 0x7f63e005c7c0 2021-03-09T09:16:46.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:46.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:46.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f7c0 con 0x7f63e005c7c0 2021-03-09T09:16:46.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:16:46.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.279+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:46.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.281+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:16:46.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.281+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:46.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.281+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=4650 cs=0 l=1 rev1=1 rx=0x7f63e00137c0 tx=0x7f63f0065890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:46.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0021760 con 0x7f63e005c7c0 2021-03-09T09:16:46.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.282+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:46.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.282+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:46.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.282+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:46.391 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:46.392 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:46.392 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:46.392 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:46.394 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:46.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0021760 con 0x7f63e005c7c0 2021-03-09T09:16:46.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.284+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:46.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.284+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:16:46.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e005c7c0 2021-03-09T09:16:46.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:46.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:46.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:47.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:47.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:47.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:47.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:48.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:48.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:48.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:48.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:49.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:49.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:49.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:49.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:49.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:49.668+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:50.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:50.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:50.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:50.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:51.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:16:51.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:51.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=4650 cs=0 l=1 rev1=1 rx=0x7f63e00137c0 tx=0x7f63f0065890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:51.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=4650 cs=0 l=1 rev1=1 rx=0x7f63e00137c0 tx=0x7f63f0065890).stop 2021-03-09T09:16:51.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:51.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:51.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:51.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=CLOSED pgs=4650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:51.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:16:51.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:51.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:51.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:51.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:51.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:51.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:51.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:51.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022a80 con 0x7f63e00070a0 2021-03-09T09:16:51.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:51.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:16:51.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:51.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f670 con 0x7f6400146050 2021-03-09T09:16:51.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:51.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:51.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:51.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:51.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.280+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:51.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.281+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:51.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.281+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:16:51.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.281+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:51.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.281+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:51.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.281+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:51.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.281+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:51.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.281+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:51.291 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:51.281+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12464 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:51.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:51.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:51.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:51.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:51.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:51.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:16:51.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:16:51.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:51.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:51.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0030420 con 0x7f6400146050 2021-03-09T09:16:51.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:51.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:51.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001ab0 con 0x7f6400146050 2021-03-09T09:16:51.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:16:51.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.282+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:51.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.283+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e100 secure :-1 s=READY pgs=4662 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc01b160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:51.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.283+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:16:51.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.283+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:51.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:16:51.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.284+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:51.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.284+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:51.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.284+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:51.297 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:51.297 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:51.298 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:51.299 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:51.299 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:51.299 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:51.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.285+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:16:51.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.285+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.285+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:16:51.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:16:51.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:51.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:51.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:52.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:52.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:52.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:52.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:52.555+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:53.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:53.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:53.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:53.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:54.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:54.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:54.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:54.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:54.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:54.669+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:16:55.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:55.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:55.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:55.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:56.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:16:56.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:16:56.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e100 secure :-1 s=READY pgs=4662 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc01b160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:16:56.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e100 secure :-1 s=READY pgs=4662 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc01b160).stop 2021-03-09T09:16:56.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:16:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005e100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:16:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005e100 unknown :-1 s=CLOSED pgs=4662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:16:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:16:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:16:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:16:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:16:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:16:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068d50 con 0x7f63e00070a0 2021-03-09T09:16:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a6d0 con 0x7f6400145bd0 2021-03-09T09:16:56.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:16:56.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017710 con 0x7f63e005c7c0 2021-03-09T09:16:56.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:16:56.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:16:56.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:56.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:16:56.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:16:56.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:16:56.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.282+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:16:56.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:16:56.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:16:56.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:16:56.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:16:56.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:16:56.293 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12500 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:16:56.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:16:56.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:56.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:56.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:16:56.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:16:56.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:16:56.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:16:56.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:16:56.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:56.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012960 con 0x7f63e005c7c0 2021-03-09T09:16:56.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:16:56.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:16:56.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025c60 con 0x7f63e005c7c0 2021-03-09T09:16:56.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:16:56.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:56.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.283+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=4671 cs=0 l=1 rev1=1 rx=0x7f63f4062e10 tx=0x7f63f4062c80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:16:56.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.284+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:16:56.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.284+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:16:56.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4023be0 con 0x7f63e005c7c0 2021-03-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.284+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.284+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.284+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:16:56.300 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:56.300 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:16:56.300 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:16:56.300 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:16:56.300 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:16:56.301 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:16:56.301 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:16:56.301 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:16:56.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.285+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4023d40 con 0x7f63e005c7c0 2021-03-09T09:16:56.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.285+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:16:56.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.285+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:16:56.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.287+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f63e005c7c0 2021-03-09T09:16:56.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:56.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:56.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:57.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:57.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:57.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:57.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:58.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:58.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:58.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:59.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:59.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:16:59.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:59.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:16:59.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:16:59.670+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:00.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:00.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:01.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:17:01.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:01.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=4671 cs=0 l=1 rev1=1 rx=0x7f63f4062e10 tx=0x7f63f4062c80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:01.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 secure :-1 s=READY pgs=4671 cs=0 l=1 rev1=1 rx=0x7f63f4062e10 tx=0x7f63f4062c80).stop 2021-03-09T09:17:01.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:01.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:01.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:01.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e002b010 unknown :-1 s=CLOSED pgs=4671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:01.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:17:01.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:01.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:01.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:01.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:01.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:01.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:01.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:01.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e360 con 0x7f63e00070a0 2021-03-09T09:17:01.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:01.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018f30 con 0x7f6400145bd0 2021-03-09T09:17:01.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:01.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:17:01.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:01.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:01.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:01.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:01.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:01.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:01.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:17:01.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:01.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.284+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:01.294 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:01.285+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12540 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:01.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.285+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:01.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.285+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:01.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.285+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:01.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:01.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:01.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:01.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:01.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:01.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:17:01.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:17:01.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:01.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:01.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:17:01.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:01.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:01.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0030420 con 0x7f6400146050 2021-03-09T09:17:01.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:17:01.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.286+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:01.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.287+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a6d0 secure :-1 s=READY pgs=4691 cs=0 l=1 rev1=1 rx=0x7f63fc071110 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:01.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.287+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:17:01.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.287+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:01.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.289+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02bba0 con 0x7f6400146050 2021-03-09T09:17:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.289+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.289+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.289+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:01.302 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:01.302 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:01.302 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:01.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.289+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02bba0 con 0x7f6400146050 2021-03-09T09:17:01.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.289+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:01.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.289+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:17:01.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:17:01.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:01.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:01.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:02.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:02.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:02.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:02.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:02.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:02.556+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:03.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:03.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:03.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:03.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:04.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:04.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:04.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:04.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:04.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:04.671+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:05.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:05.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:05.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:05.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:06.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:17:06.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:06.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a6d0 secure :-1 s=READY pgs=4691 cs=0 l=1 rev1=1 rx=0x7f63fc071110 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:06.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a6d0 secure :-1 s=READY pgs=4691 cs=0 l=1 rev1=1 rx=0x7f63fc071110 tx=0x7f63fc04a740).stop 2021-03-09T09:17:06.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:06.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:06.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e007a6d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:06.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e007a6d0 unknown :-1 s=CLOSED pgs=4691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:06.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:17:06.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:06.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:06.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:06.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:06.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:06.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:06.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:06.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023d10 con 0x7f63e00070a0 2021-03-09T09:17:06.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:06.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400145bd0 2021-03-09T09:17:06.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:06.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a6d0 con 0x7f63e005c7c0 2021-03-09T09:17:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:06.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:06.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:06.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:17:06.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:06.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:06.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:06.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:06.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.286+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:06.297 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:06.287+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12576 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:06.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:06.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:06.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:06.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:06.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:06.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:17:06.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:17:06.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:06.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:06.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0008cd0 con 0x7f63e005c7c0 2021-03-09T09:17:06.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:06.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:06.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029410 con 0x7f63e005c7c0 2021-03-09T09:17:06.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:17:06.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:06.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cbd0 secure :-1 s=READY pgs=4703 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:06.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:17:06.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.288+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:06.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.291+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000ec30 con 0x7f63e005c7c0 2021-03-09T09:17:06.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.291+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:06.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.291+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:06.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.291+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:06.307 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:06.307 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:06.307 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:06.308 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:06.308 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:06.309 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:06.309 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:06.309 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:06.309 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:06.310 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:06.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000ec30 con 0x7f63e005c7c0 2021-03-09T09:17:06.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.293+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:06.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.293+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:17:06.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d3a0 con 0x7f63e005c7c0 2021-03-09T09:17:06.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:06.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:06.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:07.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:07.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:07.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:07.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:08.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:08.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:08.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:08.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:09.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:09.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:09.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:09.672+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:10.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:10.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:10.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:10.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:11.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:17:11.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:11.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cbd0 secure :-1 s=READY pgs=4703 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:11.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cbd0 secure :-1 s=READY pgs=4703 cs=0 l=1 rev1=1 rx=0x7f63f0021040 tx=0x7f63f0000c30).stop 2021-03-09T09:17:11.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:11.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007cbd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:11.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007cbd0 unknown :-1 s=CLOSED pgs=4703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:11.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:17:11.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:11.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:17:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a8f0 con 0x7f6400145bd0 2021-03-09T09:17:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f870 con 0x7f6400146050 2021-03-09T09:17:11.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:11.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:11.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:11.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:11.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.288+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:11.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.289+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:11.303 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:11.289+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12616 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.289+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:17:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.289+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:11.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.289+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:11.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:11.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:11.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:11.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:11.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:11.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:11.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:11.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:11.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:17:11.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:17:11.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:11.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:11.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d60 con 0x7f6400146050 2021-03-09T09:17:11.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:11.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:11.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:17:11.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:17:11.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.290+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:11.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.291+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069e80 secure :-1 s=READY pgs=4718 cs=0 l=1 rev1=1 rx=0x7f63e00242a0 tx=0x7f63fc0039e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:11.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.291+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:17:11.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.291+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:11.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012ec0 con 0x7f6400146050 2021-03-09T09:17:11.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.292+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:11.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.292+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:11.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.292+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:11.319 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:11.319 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:11.320 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:11.320 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:11.321 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:11.321 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:11.321 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02bba0 con 0x7f6400146050 2021-03-09T09:17:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.292+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:11.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.292+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:17:11.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.294+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc033290 con 0x7f6400146050 2021-03-09T09:17:11.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:11.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:12.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:12.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:12.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:12.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:12.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:12.557+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:13.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:13.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:13.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:13.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:14.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:14.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:14.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:14.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:14.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:14.673+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:15.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:15.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:15.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:15.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:16.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:17:16.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:16.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069e80 secure :-1 s=READY pgs=4718 cs=0 l=1 rev1=1 rx=0x7f63e00242a0 tx=0x7f63fc0039e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:16.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069e80 secure :-1 s=READY pgs=4718 cs=0 l=1 rev1=1 rx=0x7f63e00242a0 tx=0x7f63fc0039e0).stop 2021-03-09T09:17:16.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:16.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:16.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0069e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:16.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0069e80 unknown :-1 s=CLOSED pgs=4718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:16.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:17:16.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:16.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:16.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:16.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:16.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:16.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:16.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:16.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:17:16.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:16.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c890 con 0x7f6400145bd0 2021-03-09T09:17:16.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:16.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023d10 con 0x7f63e005c7c0 2021-03-09T09:17:16.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:16.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:16.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:16.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:16.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.290+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:16.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:16.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:17:16.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:16.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:16.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:16.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:16.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:16.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:16.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:16.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:16.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:16.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:16.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:17:16.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:17:16.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:16.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:16.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cc60 con 0x7f63e005c7c0 2021-03-09T09:17:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012960 con 0x7f63e005c7c0 2021-03-09T09:17:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.292+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:17:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.292+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:16.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.292+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:17:16.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.292+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:16.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.292+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4726 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40051f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:16.306 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:16.291+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12652 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:16.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.294+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e005c7c0 2021-03-09T09:17:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.294+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.294+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.294+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:16.309 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:16.309 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:16.309 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:16.309 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:16.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.294+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40091e0 con 0x7f63e005c7c0 2021-03-09T09:17:16.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.294+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:16.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.294+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:17:16.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037070 con 0x7f63e005c7c0 2021-03-09T09:17:16.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:16.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:16.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:17.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:17.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:17.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:17.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:18.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:18.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:18.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:18.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:19.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:19.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:19.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:19.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:19.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:19.674+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:20.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:20.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:20.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:20.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:21.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:17:21.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:21.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4726 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40051f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:21.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4726 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f40051f0).stop 2021-03-09T09:17:21.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:21.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:21.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:21.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:21.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:17:21.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:21.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:21.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:21.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:21.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:21.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:21.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:21.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00107b0 con 0x7f63e00070a0 2021-03-09T09:17:21.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:21.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f870 con 0x7f6400145bd0 2021-03-09T09:17:21.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:21.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:17:21.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:21.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:21.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:21.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:21.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:21.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:21.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:17:21.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:21.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.293+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:21.303 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:21.294+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12692 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:21.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.294+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:21.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.294+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:21.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.294+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:21.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:21.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:21.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:21.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:21.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:21.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:17:21.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:17:21.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:21.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:21.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001ab0 con 0x7f6400146050 2021-03-09T09:17:21.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:21.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:21.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cc60 con 0x7f6400146050 2021-03-09T09:17:21.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:17:21.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:21.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012a00 secure :-1 s=READY pgs=4743 cs=0 l=1 rev1=1 rx=0x7f63f4000c40 tx=0x7f63f4022900).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:21.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:17:21.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.295+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:21.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.296+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022b30 con 0x7f6400146050 2021-03-09T09:17:21.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.296+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:21.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.296+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:21.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.296+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:21.314 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:21.314 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:21.314 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:21.314 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:21.314 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:21.315 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:21.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022b30 con 0x7f6400146050 2021-03-09T09:17:21.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.299+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:21.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.299+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:17:21.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f6400146050 2021-03-09T09:17:21.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:21.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:21.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:22.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:22.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:22.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:22.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:22.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:22.559+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:23.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:23.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:23.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:23.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:24.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:24.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:24.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:24.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:24.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:24.675+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:25.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:25.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:25.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:25.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:26.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:17:26.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:26.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012a00 secure :-1 s=READY pgs=4743 cs=0 l=1 rev1=1 rx=0x7f63f4000c40 tx=0x7f63f4022900).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:26.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012a00 secure :-1 s=READY pgs=4743 cs=0 l=1 rev1=1 rx=0x7f63f4000c40 tx=0x7f63f4022900).stop 2021-03-09T09:17:26.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:26.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:26.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:26.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012a00 unknown :-1 s=CLOSED pgs=4743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:26.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:17:26.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:26.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:26.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:26.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:26.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:26.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:26.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:26.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f1b0 con 0x7f63e00070a0 2021-03-09T09:17:26.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:26.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f6400145bd0 2021-03-09T09:17:26.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:26.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e005c7c0 2021-03-09T09:17:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.295+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:26.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.296+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:26.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.296+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:26.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.296+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:26.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.296+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:17:26.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.296+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:26.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.296+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:26.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.296+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:26.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.296+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:26.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.296+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:26.305 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12728 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:26.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:26.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:26.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:26.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:26.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:26.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:17:26.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:17:26.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:26.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:26.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a60 con 0x7f63e005c7c0 2021-03-09T09:17:26.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:26.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:26.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002d60 con 0x7f63e005c7c0 2021-03-09T09:17:26.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:17:26.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.297+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:26.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.298+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:17:26.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.298+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:26.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.298+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4758 cs=0 l=1 rev1=1 rx=0x7f63e000bba0 tx=0x7f63fc039f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:26.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f63e005c7c0 2021-03-09T09:17:26.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.300+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:26.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.300+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:26.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.300+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:26.312 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:26.312 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:26.312 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:26.313 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:26.313 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:26.313 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:26.313 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:26.313 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:26.314 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:26.314 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:26.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.302+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071140 con 0x7f63e005c7c0 2021-03-09T09:17:26.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.302+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:26.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.302+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:17:26.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f63e005c7c0 2021-03-09T09:17:26.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:26.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:26.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:27.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:27.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:27.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:27.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:28.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:28.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:28.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:28.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:29.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:29.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:29.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:29.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:29.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:29.676+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:30.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:30.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:30.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:30.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:31.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:17:31.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:31.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4758 cs=0 l=1 rev1=1 rx=0x7f63e000bba0 tx=0x7f63fc039f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:31.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4758 cs=0 l=1 rev1=1 rx=0x7f63e000bba0 tx=0x7f63fc039f30).stop 2021-03-09T09:17:31.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:31.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:31.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:31.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:31.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:17:31.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:31.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:31.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:31.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:31.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:31.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:31.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:31.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e00070a0 2021-03-09T09:17:31.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:31.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bdc0 con 0x7f6400145bd0 2021-03-09T09:17:31.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:31.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013480 con 0x7f6400146050 2021-03-09T09:17:31.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.297+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:31.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.298+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:31.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.298+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:31.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.298+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:31.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.298+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:31.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.298+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:31.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.298+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:17:31.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.298+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:31.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.298+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:31.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:31.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:31.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:31.316 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12768 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:31.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:31.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:31.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:31.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:31.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:31.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:17:31.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.299+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:17:31.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.300+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:31.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.300+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:31.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.300+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a8d0 con 0x7f6400146050 2021-03-09T09:17:31.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.300+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:31.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.300+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:31.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.300+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0008cd0 con 0x7f6400146050 2021-03-09T09:17:31.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.300+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:17:31.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.300+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:31.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.301+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:17:31.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.301+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:31.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 secure :-1 s=READY pgs=4774 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004cf10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:31.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.302+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f6400146050 2021-03-09T09:17:31.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.302+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:31.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.302+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:31.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.302+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:31.329 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:31.332 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:31.332 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:31.333 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:31.334 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:31.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f6400146050 2021-03-09T09:17:31.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.304+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:31.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f6400146050 2021-03-09T09:17:31.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.304+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:17:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:31.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:32.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:32.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:32.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:32.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:32.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:32.561+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:33.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:33.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:33.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:33.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:34.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:34.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:34.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:34.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:34.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:34.677+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:35.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:35.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:35.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:35.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:36.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:17:36.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:36.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 secure :-1 s=READY pgs=4774 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004cf10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:36.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 secure :-1 s=READY pgs=4774 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004cf10).stop 2021-03-09T09:17:36.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:36.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:36.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00114d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:36.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 unknown :-1 s=CLOSED pgs=4774 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:36.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:17:36.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:36.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:36.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:36.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:36.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:36.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:36.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:36.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066690 con 0x7f63e00070a0 2021-03-09T09:17:36.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:36.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:17:36.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:36.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f1b0 con 0x7f63e005c7c0 2021-03-09T09:17:36.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:36.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:36.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:36.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:36.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:36.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:17:36.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:36.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:36.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.300+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:36.309 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:36.301+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12804 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:36.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.301+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:36.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.301+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:36.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.301+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:36.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:36.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:36.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:36.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:36.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:36.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:17:36.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:17:36.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:36.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:36.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f0e0 con 0x7f63e005c7c0 2021-03-09T09:17:36.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:36.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:36.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a8d0 con 0x7f63e005c7c0 2021-03-09T09:17:36.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:17:36.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:36.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:17:36.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4784 cs=0 l=1 rev1=1 rx=0x7f63f0000ca0 tx=0x7f63f001b490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:36.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.302+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:36.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00157f0 con 0x7f63e005c7c0 2021-03-09T09:17:36.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.304+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:36.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.304+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:36.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.304+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:36.317 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:36.317 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:36.317 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:36.318 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:36.318 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:36.318 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:36.318 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:36.318 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:36.319 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:36.319 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:36.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0048730 con 0x7f63e005c7c0 2021-03-09T09:17:36.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.305+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:36.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.305+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:17:36.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001cb70 con 0x7f63e005c7c0 2021-03-09T09:17:36.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:36.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:36.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:37.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:37.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:37.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:37.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:38.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:38.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:38.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:38.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:39.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:39.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:39.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:39.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:39.678+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:40.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:40.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:40.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:40.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:41.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:17:41.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:41.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4784 cs=0 l=1 rev1=1 rx=0x7f63f0000ca0 tx=0x7f63f001b490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:41.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4784 cs=0 l=1 rev1=1 rx=0x7f63f0000ca0 tx=0x7f63f001b490).stop 2021-03-09T09:17:41.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:41.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:41.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:41.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:41.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:17:41.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:41.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:41.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:41.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:41.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:41.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f63e00070a0 2021-03-09T09:17:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023d10 con 0x7f6400145bd0 2021-03-09T09:17:41.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068f50 con 0x7f6400146050 2021-03-09T09:17:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.301+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:41.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.302+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:41.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.302+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:41.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.302+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:41.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.302+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:41.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.302+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.302+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:17:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.302+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:41.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.302+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:41.311 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:41.303+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12844 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.303+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.303+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.303+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:41.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:41.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:17:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:17:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:41.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cc60 con 0x7f6400146050 2021-03-09T09:17:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a60 con 0x7f6400146050 2021-03-09T09:17:41.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:17:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.304+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.306+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014340 secure :-1 s=READY pgs=4804 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc013300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.307+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:17:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.307+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:41.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f6400146050 2021-03-09T09:17:41.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.308+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:41.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.308+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:41.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.308+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:41.317 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:41.318 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:41.319 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:41.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f6400146050 2021-03-09T09:17:41.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.309+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:41.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.309+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:17:41.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:17:41.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:41.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:41.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:42.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:42.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:42.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:42.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:42.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:42.562+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:43.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:43.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:43.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:43.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:44.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:44.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:44.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:44.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:44.679+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:45.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:45.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:45.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:45.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:46.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:17:46.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:46.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014340 secure :-1 s=READY pgs=4804 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc013300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:46.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014340 secure :-1 s=READY pgs=4804 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc013300).stop 2021-03-09T09:17:46.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:46.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:46.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:46.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014340 unknown :-1 s=CLOSED pgs=4804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:46.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:17:46.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:46.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:46.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:46.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:46.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:46.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:46.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:46.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c650 con 0x7f63e00070a0 2021-03-09T09:17:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:17:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:46.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c890 con 0x7f63e005c7c0 2021-03-09T09:17:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:46.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:46.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:46.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:46.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:17:46.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:46.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.304+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:46.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:46.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:46.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:46.315 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12880 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:46.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:46.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:46.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:46.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:46.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:46.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:17:46.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:17:46.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:46.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:46.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029470 con 0x7f63e005c7c0 2021-03-09T09:17:46.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:46.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:46.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cc60 con 0x7f63e005c7c0 2021-03-09T09:17:46.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:17:46.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.305+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:46.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.306+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:17:46.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.306+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4809 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc066510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:46.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.306+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03c350 con 0x7f63e005c7c0 2021-03-09T09:17:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.307+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.307+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.307+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:46.321 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:46.322 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:46.322 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:46.322 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:46.322 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:46.323 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:46.323 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:46.323 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:46.323 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:46.324 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:46.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03c170 con 0x7f63e005c7c0 2021-03-09T09:17:46.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.307+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:46.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.307+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:17:46.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e005c7c0 2021-03-09T09:17:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:46.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:46.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:47.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:47.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:47.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:47.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:48.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:48.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:48.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:48.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:49.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:49.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:49.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:49.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:49.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:49.680+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:50.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:50.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:50.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:50.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:51.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.304+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:17:51.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.304+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:51.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.304+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4809 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc066510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:51.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.304+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4809 cs=0 l=1 rev1=1 rx=0x7f63fc058610 tx=0x7f63fc066510).stop 2021-03-09T09:17:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:51.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:51.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:51.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:17:51.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:51.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:51.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:51.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:51.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:51.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:51.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:51.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f63e00070a0 2021-03-09T09:17:51.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:51.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018dd0 con 0x7f6400145bd0 2021-03-09T09:17:51.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:51.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029940 con 0x7f6400146050 2021-03-09T09:17:51.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:51.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:51.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.305+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:51.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:51.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:51.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:51.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:17:51.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:51.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:51.315 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12919 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:51.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:51.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:51.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.306+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:51.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:51.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:51.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:51.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:51.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:51.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:17:51.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:17:51.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:51.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:51.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ef50 con 0x7f6400146050 2021-03-09T09:17:51.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:51.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:51.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029470 con 0x7f6400146050 2021-03-09T09:17:51.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:17:51.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:51.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:17:51.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:51.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.307+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 secure :-1 s=READY pgs=4823 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:51.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013370 con 0x7f6400146050 2021-03-09T09:17:51.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.308+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:51.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.308+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:51.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.308+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:51.322 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:51.323 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:51.323 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:51.323 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:51.323 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:51.324 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:51.324 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:51.324 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:51.324 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:51.324 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:51.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013370 con 0x7f6400146050 2021-03-09T09:17:51.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.308+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:51.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.308+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:17:51.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03c170 con 0x7f6400146050 2021-03-09T09:17:51.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:51.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:51.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:52.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:52.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:52.562+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:53.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:53.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:53.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:53.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:54.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:54.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:54.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:54.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:54.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:54.680+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:17:55.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:55.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:55.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:55.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:56.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.306+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:17:56.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.306+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:17:56.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.306+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 secure :-1 s=READY pgs=4823 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:17:56.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.306+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 secure :-1 s=READY pgs=4823 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc009a70).stop 2021-03-09T09:17:56.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:56.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:17:56.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00114d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:17:56.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 unknown :-1 s=CLOSED pgs=4823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:17:56.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:17:56.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:56.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:17:56.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:56.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:17:56.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:17:56.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:17:56.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:56.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b230 con 0x7f63e00070a0 2021-03-09T09:17:56.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:56.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068f50 con 0x7f6400145bd0 2021-03-09T09:17:56.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:17:56.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078270 con 0x7f63e005c7c0 2021-03-09T09:17:56.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:17:56.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:17:56.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:17:56.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:56.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:17:56.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:17:56.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:17:56.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:17:56.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:17:56.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:17:56.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:17:56.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.307+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:17:56.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:17:56.320 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12956 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:17:56.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:56.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:56.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:17:56.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:17:56.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:17:56.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:17:56.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:17:56.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:56.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a60 con 0x7f63e005c7c0 2021-03-09T09:17:56.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:17:56.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:17:56.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001ab0 con 0x7f63e005c7c0 2021-03-09T09:17:56.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:17:56.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.308+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:56.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.309+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:17:56.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.309+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4837 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f4003b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:17:56.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.309+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:17:56.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63e005c7c0 2021-03-09T09:17:56.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.310+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:17:56.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.310+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:17:56.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.310+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:17:56.328 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:17:56.328 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:17:56.328 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:17:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:17:56.329 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:17:56.330 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:17:56.330 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:17:56.330 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:17:56.330 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:17:56.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f63e005c7c0 2021-03-09T09:17:56.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.310+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:17:56.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.310+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:17:56.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:17:56.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:56.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:56.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:57.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:57.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:57.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:57.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:58.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:58.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:58.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:58.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:59.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:59.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:17:59.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:59.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:17:59.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:17:59.681+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:00.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:00.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:00.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:00.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:01.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:18:01.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:01.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4837 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f4003b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:01.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4837 cs=0 l=1 rev1=1 rx=0x7f63f4009480 tx=0x7f63f4003b40).stop 2021-03-09T09:18:01.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:01.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:01.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:01.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:01.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:18:01.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:01.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:01.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:01.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:01.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:01.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:01.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:01.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:18:01.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:01.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c890 con 0x7f6400145bd0 2021-03-09T09:18:01.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:01.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f6400146050 2021-03-09T09:18:01.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:01.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:01.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:01.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:01.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:01.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:01.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:18:01.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:01.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:01.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.309+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:01.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:01.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:01.319 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 12996 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:01.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:01.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:01.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:01.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:01.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:01.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:18:01.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:18:01.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:01.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:01.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068d50 con 0x7f6400146050 2021-03-09T09:18:01.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:01.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:01.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.310+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ef50 con 0x7f6400146050 2021-03-09T09:18:01.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.311+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:18:01.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.311+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:01.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 secure :-1 s=READY pgs=4850 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:01.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.311+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:18:01.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.311+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:01.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.312+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02bdd0 con 0x7f6400146050 2021-03-09T09:18:01.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.312+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:01.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.312+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:01.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.312+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:01.326 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:01.326 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:01.326 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:01.327 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:01.327 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:01.327 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:01.327 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:01.327 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:01.328 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:01.328 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:01.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02bba0 con 0x7f6400146050 2021-03-09T09:18:01.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.313+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:01.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.313+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:18:01.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.315+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021300 con 0x7f6400146050 2021-03-09T09:18:01.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:01.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:01.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:02.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:02.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:02.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:02.564+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:03.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:03.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:03.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:03.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:04.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:04.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:04.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:04.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:04.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:04.682+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:05.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:05.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:05.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:05.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:06.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:18:06.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:06.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 secure :-1 s=READY pgs=4850 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:06.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 secure :-1 s=READY pgs=4850 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc003a50).stop 2021-03-09T09:18:06.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:06.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:06.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00114d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:06.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00114d0 unknown :-1 s=CLOSED pgs=4850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:06.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:18:06.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:06.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:06.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:06.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:06.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:06.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:06.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:06.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a410 con 0x7f63e00070a0 2021-03-09T09:18:06.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:06.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:18:06.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:06.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63e005c7c0 2021-03-09T09:18:06.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:06.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:06.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:06.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:06.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:06.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:06.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:18:06.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:06.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.311+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:06.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:06.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:06.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:06.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:06.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:06.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:06.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:06.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:06.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:18:06.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:18:06.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:06.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:06.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a8d0 con 0x7f63e005c7c0 2021-03-09T09:18:06.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:06.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:06.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068d50 con 0x7f63e005c7c0 2021-03-09T09:18:06.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:18:06.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:06.327 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:06.312+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13032 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:06.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.313+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:18:06.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.313+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:06.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.313+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4864 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc046ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:06.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00b5d0 con 0x7f63e005c7c0 2021-03-09T09:18:06.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.314+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:06.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.314+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:06.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.314+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:06.329 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:06.329 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:06.330 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:06.330 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:06.330 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:06.330 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:06.331 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:06.331 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:06.331 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:06.331 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:06.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.315+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02bd60 con 0x7f63e005c7c0 2021-03-09T09:18:06.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.315+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:06.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.315+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:18:06.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02b9c0 con 0x7f63e005c7c0 2021-03-09T09:18:06.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:06.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:06.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:07.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:07.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:07.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:07.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:08.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:08.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:08.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:08.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:09.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:09.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:09.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:09.683+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:10.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:10.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:10.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:10.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:11.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:18:11.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:11.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4864 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc046ae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:11.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4864 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc046ae0).stop 2021-03-09T09:18:11.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:11.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:18:11.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:11.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:11.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:11.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:11.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:11.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:11.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:11.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015b40 con 0x7f63e00070a0 2021-03-09T09:18:11.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:11.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:18:11.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:11.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066450 con 0x7f6400146050 2021-03-09T09:18:11.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:11.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:11.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:11.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.313+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:11.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.314+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:11.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.314+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:18:11.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.314+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:11.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.314+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:11.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.314+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:11.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.314+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:11.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.314+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:11.324 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:11.314+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13072 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:11.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:11.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:11.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:11.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:11.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:11.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:18:11.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:18:11.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:11.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:11.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:18:11.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:11.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:11.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f0e0 con 0x7f6400146050 2021-03-09T09:18:11.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:18:11.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.315+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:11.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.316+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d170 secure :-1 s=READY pgs=4879 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0009580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:11.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.316+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:18:11.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.316+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:11.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00158c0 con 0x7f6400146050 2021-03-09T09:18:11.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.317+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:11.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.317+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:11.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.317+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:11.331 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:11.331 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:11.331 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:11.332 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:11.333 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:11.333 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:11.333 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:11.333 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:11.334 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:11.334 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:11.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:18:11.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.318+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:11.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.318+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:18:11.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.320+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001ada0 con 0x7f6400146050 2021-03-09T09:18:11.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:11.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:11.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:12.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:12.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:12.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:12.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:12.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:12.565+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:13.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:13.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:13.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:13.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:14.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:14.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:14.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:14.684+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:15.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:15.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:15.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:15.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:16.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:18:16.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:16.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d170 secure :-1 s=READY pgs=4879 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0009580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:16.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d170 secure :-1 s=READY pgs=4879 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0009580).stop 2021-03-09T09:18:16.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:16.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:16.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006d170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:16.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006d170 unknown :-1 s=CLOSED pgs=4879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:16.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:18:16.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:16.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:16.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:16.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:16.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:16.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:16.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:16.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a410 con 0x7f63e00070a0 2021-03-09T09:18:16.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:16.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400145bd0 2021-03-09T09:18:16.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:16.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001dcf0 con 0x7f63e005c7c0 2021-03-09T09:18:16.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:16.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:16.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:16.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:16.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.315+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:16.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.316+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:16.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.316+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:18:16.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.316+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:16.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.316+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:16.325 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:16.316+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13108 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:16.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.316+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:16.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.316+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:16.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.316+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:16.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.316+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:16.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:16.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:16.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:16.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:16.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:18:16.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:18:16.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:16.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:16.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067330 con 0x7f63e005c7c0 2021-03-09T09:18:16.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:16.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:16.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:18:16.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:18:16.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:16.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:18:16.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:16.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4893 cs=0 l=1 rev1=1 rx=0x7f63f0065e60 tx=0x7f63f0065f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:16.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0048960 con 0x7f63e005c7c0 2021-03-09T09:18:16.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.318+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:16.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.318+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:16.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.318+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:16.332 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:16.332 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:16.333 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:16.333 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:16.333 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:16.333 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:16.334 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:16.334 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:16.334 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:16.335 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0048ac0 con 0x7f63e005c7c0 2021-03-09T09:18:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.318+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:16.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.319+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:18:16.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.319+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0048c30 con 0x7f63e005c7c0 2021-03-09T09:18:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:16.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:16.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:17.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:17.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:17.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:17.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:18.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:18.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:18.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:18.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:19.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:19.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:19.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:19.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:19.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:19.685+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:20.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:20.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:20.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:20.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:21.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.316+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:18:21.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.316+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:21.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.316+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4893 cs=0 l=1 rev1=1 rx=0x7f63f0065e60 tx=0x7f63f0065f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:21.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.316+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4893 cs=0 l=1 rev1=1 rx=0x7f63f0065e60 tx=0x7f63f0065f40).stop 2021-03-09T09:18:21.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.316+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:21.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.316+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:21.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:21.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.316+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:21.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.316+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:18:21.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:21.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:21.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:21.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:21.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:21.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:21.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:21.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:18:21.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:21.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400145bd0 2021-03-09T09:18:21.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:21.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a410 con 0x7f6400146050 2021-03-09T09:18:21.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:21.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:21.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:21.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:21.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:21.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:21.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:18:21.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:21.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:21.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:21.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:21.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:21.328 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13148 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:21.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:21.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:21.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:21.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.318+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:21.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:21.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:18:21.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:18:21.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:21.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:21.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00684f0 con 0x7f6400146050 2021-03-09T09:18:21.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:21.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:21.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a8d0 con 0x7f6400146050 2021-03-09T09:18:21.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:18:21.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:21.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006dbc0 secure :-1 s=READY pgs=4905 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:21.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:18:21.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.319+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:21.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f6400146050 2021-03-09T09:18:21.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.321+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:21.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.321+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:21.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.321+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:21.335 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:21.335 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:21.335 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:21.335 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:21.336 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:21.336 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:21.336 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:21.336 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:21.336 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:21.337 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:21.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f6400146050 2021-03-09T09:18:21.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.323+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:21.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.323+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:18:21.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:18:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:21.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:21.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:22.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:22.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:22.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:22.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:22.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:22.566+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:23.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:23.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:23.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:23.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:24.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:24.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:24.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:24.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:24.686+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:25.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:25.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:25.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:25.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:26.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:18:26.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:26.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006dbc0 secure :-1 s=READY pgs=4905 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:26.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006dbc0 secure :-1 s=READY pgs=4905 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc009a70).stop 2021-03-09T09:18:26.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:26.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006dbc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006dbc0 unknown :-1 s=CLOSED pgs=4905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:18:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:26.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:26.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:26.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:26.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:26.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:26.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:18:26.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:26.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023d60 con 0x7f6400145bd0 2021-03-09T09:18:26.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:26.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e005c7c0 2021-03-09T09:18:26.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:26.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:26.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:26.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:26.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:18:26.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:26.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:26.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:26.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:26.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:26.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:26.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.319+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:26.330 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13184 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:26.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:26.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:26.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:26.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:26.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:26.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:18:26.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:18:26.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.320+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a280 con 0x7f63e005c7c0 2021-03-09T09:18:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.321+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.321+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.321+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00684f0 con 0x7f63e005c7c0 2021-03-09T09:18:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.321+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:18:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.321+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.321+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4919 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc00b650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.321+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:18:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.321+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.322+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0233d0 con 0x7f63e005c7c0 2021-03-09T09:18:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.322+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.322+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:26.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.322+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:26.337 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:26.337 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:26.337 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:26.337 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:26.338 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:26.338 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:26.338 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:26.339 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:26.339 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:26.339 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:26.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63e005c7c0 2021-03-09T09:18:26.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.323+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:26.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.323+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:18:26.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e005c7c0 2021-03-09T09:18:26.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:26.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:26.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:27.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:27.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:27.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:27.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:28.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:28.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:28.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:28.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:29.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.343+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 msgr2=0x7f63e0017850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:18:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.343+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 msgr2=0x7f63e0017850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.343+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 0x7f63e0017850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0006b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.343+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 0x7f63e0017850 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0006b30).stop 2021-03-09T09:18:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.343+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 msgr2=0x7f63e0017850 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T09:18:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 msgr2=0x7f63e0017850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.344+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0064730 0x7f63e0017850 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.344+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 0x7f63e0025840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.344+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 0x7f63e0025840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.344+0000 7f6405d52700 10 monclient: get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:18:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.344+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:18:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.344+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 0x7f63e0025840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f63f0009550 tx=0x7f63f0006b30).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:29.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:29.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:29.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:29.687+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:30.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:30.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:30.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:30.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:31.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:18:31.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:31.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4919 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc00b650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:31.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4919 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc00b650).stop 2021-03-09T09:18:31.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:31.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:31.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:31.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:31.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:18:31.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:31.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:31.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:31.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:31.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:31.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:31.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:31.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63e00070a0 2021-03-09T09:18:31.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:31.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a410 con 0x7f6400145bd0 2021-03-09T09:18:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:18:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.321+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:31.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.322+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:31.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.322+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:18:31.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.322+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:31.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.322+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:31.333 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:31.322+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13224 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:31.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:31.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:31.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:31.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:31.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:31.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:31.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:31.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:31.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:18:31.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:18:31.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:31.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:31.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005cff0 con 0x7f63e0064730 2021-03-09T09:18:31.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:31.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:31.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067330 con 0x7f63e0064730 2021-03-09T09:18:31.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:18:31.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.323+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:31.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.324+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:18:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.324+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.324+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017760 secure :-1 s=READY pgs=4936 cs=0 l=1 rev1=1 rx=0x7f63e001c600 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001b250 con 0x7f63e0064730 2021-03-09T09:18:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.325+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.325+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:31.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.325+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:31.342 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:31.343 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:31.343 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:31.343 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:31.343 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:31.344 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:31.344 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:31.344 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:31.344 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:31.345 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:31.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.326+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001b250 con 0x7f63e0064730 2021-03-09T09:18:31.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.326+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:31.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.326+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:18:31.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004a070 con 0x7f63e0064730 2021-03-09T09:18:31.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:31.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:31.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:32.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:32.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:32.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:32.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:32.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:32.568+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:33.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:33.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:33.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:33.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:34.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:34.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:34.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:34.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:34.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:34.688+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:35.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:35.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:35.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:35.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:36.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0017760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:18:36.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0017760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:36.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017760 secure :-1 s=READY pgs=4936 cs=0 l=1 rev1=1 rx=0x7f63e001c600 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:36.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017760 secure :-1 s=READY pgs=4936 cs=0 l=1 rev1=1 rx=0x7f63e001c600 tx=0x7f63f0013b80).stop 2021-03-09T09:18:36.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:36.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:36.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0017760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:36.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0017760 unknown :-1 s=CLOSED pgs=4936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:36.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:18:36.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:36.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:36.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:36.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:36.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:36.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:36.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:36.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066450 con 0x7f63e00070a0 2021-03-09T09:18:36.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:36.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a410 con 0x7f6400145bd0 2021-03-09T09:18:36.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001eca0 con 0x7f63e005c7c0 2021-03-09T09:18:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.323+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.324+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:36.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.324+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:18:36.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.324+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.324+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.324+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.324+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.324+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:36.346 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:36.324+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13260 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:36.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:36.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:36.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:18:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:18:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010db0 con 0x7f63e005c7c0 2021-03-09T09:18:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a280 con 0x7f63e005c7c0 2021-03-09T09:18:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:18:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.325+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.326+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4950 cs=0 l=1 rev1=1 rx=0x7f63e002bad0 tx=0x7f63fc046bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.326+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:18:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.326+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d740 con 0x7f63e005c7c0 2021-03-09T09:18:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.327+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.327+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.327+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:36.356 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:36.356 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:36.357 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:36.357 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:36.357 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:36.358 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:36.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63e005c7c0 2021-03-09T09:18:36.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.328+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:36.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.328+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:18:36.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.330+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f63e005c7c0 2021-03-09T09:18:36.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:36.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:36.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:37.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:37.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:37.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:37.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:38.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:38.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:38.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:38.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:39.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:39.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:39.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:39.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:39.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:39.689+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:40.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:40.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:40.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:40.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:41.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:18:41.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:41.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4950 cs=0 l=1 rev1=1 rx=0x7f63e002bad0 tx=0x7f63fc046bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:41.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4950 cs=0 l=1 rev1=1 rx=0x7f63e002bad0 tx=0x7f63fc046bc0).stop 2021-03-09T09:18:41.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:41.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:41.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:41.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:41.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:18:41.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0067fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:41.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:41.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:41.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:41.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:41.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:41.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:41.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:18:41.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:41.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015b40 con 0x7f6400145bd0 2021-03-09T09:18:41.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:41.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005df90 con 0x7f63e0064730 2021-03-09T09:18:41.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:41.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:41.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:41.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:41.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.325+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:41.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.326+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0067fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:41.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.326+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:18:41.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.326+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:41.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.326+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:41.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.326+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:41.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.326+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:41.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.326+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:41.343 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:41.326+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13300 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:41.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:41.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:41.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:41.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:41.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:41.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:18:41.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:18:41.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:41.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:41.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067330 con 0x7f63e0064730 2021-03-09T09:18:41.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:41.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:41.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a60 con 0x7f63e0064730 2021-03-09T09:18:41.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:18:41.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:41.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:18:41.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0067fa0 secure :-1 s=READY pgs=4963 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4005c80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:41.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.327+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:41.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.329+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009350 con 0x7f63e0064730 2021-03-09T09:18:41.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.329+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:41.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.329+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:41.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.329+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:41.363 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:41.363 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:41.364 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:41.364 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:41.365 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:41.365 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:41.365 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:41.366 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:41.366 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:41.367 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:41.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0064730 2021-03-09T09:18:41.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.331+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.331+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:18:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T09:18:41.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:41.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:41.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:42.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:42.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:42.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:42.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:42.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:42.569+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:43.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:43.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:43.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:43.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:44.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:44.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:44.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:44.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:44.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:44.690+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:45.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:45.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:45.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:45.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:46.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0067fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:18:46.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0067fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:46.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0067fa0 secure :-1 s=READY pgs=4963 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4005c80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:46.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0067fa0 secure :-1 s=READY pgs=4963 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4005c80).stop 2021-03-09T09:18:46.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:46.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:46.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0067fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:46.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0067fa0 unknown :-1 s=CLOSED pgs=4963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:46.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:18:46.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:46.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:46.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:46.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:46.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:46.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:46.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:46.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:18:46.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d40 con 0x7f6400145bd0 2021-03-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f63e005c7c0 2021-03-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.327+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:46.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:46.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:18:46.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:46.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:46.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:46.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:46.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:46.343 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13336 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:46.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:46.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:46.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:46.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:46.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:46.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:18:46.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:18:46.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:46.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:46.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:18:46.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:46.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:46.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005cff0 con 0x7f63e005c7c0 2021-03-09T09:18:46.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:18:46.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.328+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:46.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.329+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:18:46.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.329+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:46.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4976 cs=0 l=1 rev1=1 rx=0x7f63f0053040 tx=0x7f63f001b330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:46.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.330+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e005c7c0 2021-03-09T09:18:46.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.330+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:46.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.330+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:46.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.330+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:46.352 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:46.352 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:46.352 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:46.352 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:46.353 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:46.353 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:46.353 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:46.353 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:46.354 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:46.354 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:46.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.330+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00157f0 con 0x7f63e005c7c0 2021-03-09T09:18:46.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.330+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:46.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.330+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:18:46.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.332+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e005c7c0 2021-03-09T09:18:46.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:46.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:46.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:47.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:47.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:47.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:47.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:48.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:48.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:48.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:48.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:49.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:49.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:49.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:49.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:49.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:49.691+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:50.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:50.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:50.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:50.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:51.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:18:51.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:51.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4976 cs=0 l=1 rev1=1 rx=0x7f63f0053040 tx=0x7f63f001b330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:51.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=4976 cs=0 l=1 rev1=1 rx=0x7f63f0053040 tx=0x7f63f001b330).stop 2021-03-09T09:18:51.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:51.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:51.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:51.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=4976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:18:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:51.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:51.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:51.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:51.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.328+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:51.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:51.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024e40 con 0x7f63e00070a0 2021-03-09T09:18:51.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:51.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00096e0 con 0x7f6400145bd0 2021-03-09T09:18:51.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:51.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d40 con 0x7f63e0064730 2021-03-09T09:18:51.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:51.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:51.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:51.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:51.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:51.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:51.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:18:51.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:51.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:51.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:51.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:51.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.329+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:51.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:51.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:51.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:51.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:51.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:51.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:18:51.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:18:51.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:51.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:51.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068da0 con 0x7f63e0064730 2021-03-09T09:18:51.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:51.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:51.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T09:18:51.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:18:51.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:51.342 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:51.330+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13367 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=4988 cs=0 l=1 rev1=1 rx=0x7f63f000eca0 tx=0x7f63f0030b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:51.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.331+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:18:51.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.331+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:51.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030e30 con 0x7f63e0064730 2021-03-09T09:18:51.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.334+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:51.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.334+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:51.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.334+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:51.345 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:51.345 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:51.345 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:51.346 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:51.346 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:51.346 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:51.346 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:51.346 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:51.347 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:51.347 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:51.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030e30 con 0x7f63e0064730 2021-03-09T09:18:51.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.334+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:51.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.334+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:18:51.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004ccc0 con 0x7f63e0064730 2021-03-09T09:18:51.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:51.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:51.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:52.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:52.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:52.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:52.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:52.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:52.570+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:53.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:53.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:53.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:53.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:54.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:54.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:54.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:54.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:54.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:54.692+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:18:55.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:55.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:55.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:55.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:56.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:18:56.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:18:56.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=4988 cs=0 l=1 rev1=1 rx=0x7f63f000eca0 tx=0x7f63f0030b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:18:56.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=4988 cs=0 l=1 rev1=1 rx=0x7f63f000eca0 tx=0x7f63f0030b10).stop 2021-03-09T09:18:56.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:56.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:18:56.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:18:56.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=4988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:18:56.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:18:56.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:56.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:18:56.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:56.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:18:56.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:18:56.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:18:56.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:56.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b590 con 0x7f63e00070a0 2021-03-09T09:18:56.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:56.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066890 con 0x7f6400145bd0 2021-03-09T09:18:56.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:18:56.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:18:56.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:18:56.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008a70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:18:56.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:56.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:18:56.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.331+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:18:56.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.332+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:18:56.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.332+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:18:56.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.332+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:18:56.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.332+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:18:56.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.332+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:18:56.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.332+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:18:56.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.332+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:18:56.342 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13412 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:18:56.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:18:56.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:56.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:56.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:18:56.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:18:56.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:18:56.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:18:56.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:18:56.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:56.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068c10 con 0x7f63e005c7c0 2021-03-09T09:18:56.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:18:56.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:18:56.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068da0 con 0x7f63e005c7c0 2021-03-09T09:18:56.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:18:56.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.333+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:56.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.334+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:18:56.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5007 cs=0 l=1 rev1=1 rx=0x7f63f005de70 tx=0x7f63f004e7c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:18:56.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.334+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:18:56.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055eb0 con 0x7f63e005c7c0 2021-03-09T09:18:56.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.335+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:18:56.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.335+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:18:56.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.335+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:18:56.349 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:18:56.349 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:18:56.349 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:56.349 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:18:56.350 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:18:56.350 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:18:56.350 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:18:56.350 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:18:56.350 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:18:56.351 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:18:56.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004e9f0 con 0x7f63e005c7c0 2021-03-09T09:18:56.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.335+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:18:56.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.335+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:18:56.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.337+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0030ce0 con 0x7f63e005c7c0 2021-03-09T09:18:56.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:56.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:56.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:57.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:57.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:57.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:57.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:58.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:58.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:58.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:58.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:59.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:59.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:18:59.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:59.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:18:59.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:18:59.693+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:00.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:00.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:00.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:00.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:01.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5007 cs=0 l=1 rev1=1 rx=0x7f63f005de70 tx=0x7f63f004e7c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:01.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5007 cs=0 l=1 rev1=1 rx=0x7f63f005de70 tx=0x7f63f004e7c0).stop 2021-03-09T09:19:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:01.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:19:01.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:01.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:01.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:01.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:01.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:01.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:01.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:01.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d40 con 0x7f63e00070a0 2021-03-09T09:19:01.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:01.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013440 con 0x7f6400145bd0 2021-03-09T09:19:01.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:01.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f360 con 0x7f63e0064730 2021-03-09T09:19:01.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:01.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.333+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:01.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.334+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.334+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:19:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.334+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:01.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.334+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:01.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.334+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:01.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.334+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:01.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.334+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:01.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:01.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:01.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:01.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:01.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:01.346 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13450 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:01.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:19:01.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:19:01.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:01.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:01.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00109f0 con 0x7f63e0064730 2021-03-09T09:19:01.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:01.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:01.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067330 con 0x7f63e0064730 2021-03-09T09:19:01.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:19:01.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:01.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=5018 cs=0 l=1 rev1=1 rx=0x7f63f405b9d0 tx=0x7f63f402af00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:01.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:19:01.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.335+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:01.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.336+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f63e0064730 2021-03-09T09:19:01.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.336+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:01.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.336+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:01.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.336+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:01.351 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:01.351 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:01.351 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:01.351 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:01.352 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:01.353 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:01.353 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:01.353 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:01.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.337+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f63e0064730 2021-03-09T09:19:01.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.337+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:01.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.337+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:19:01.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T09:19:01.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:01.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:02.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:02.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:02.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:02.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:02.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:02.571+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:03.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:03.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:03.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:03.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:04.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:04.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:04.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:04.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:04.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:04.694+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:05.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:05.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:05.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:05.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:06.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:06.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:06.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=5018 cs=0 l=1 rev1=1 rx=0x7f63f405b9d0 tx=0x7f63f402af00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:06.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=5018 cs=0 l=1 rev1=1 rx=0x7f63f405b9d0 tx=0x7f63f402af00).stop 2021-03-09T09:19:06.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:06.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:06.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:06.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=5018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:06.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:19:06.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:06.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:06.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:06.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:06.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:06.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:06.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:06.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012590 con 0x7f63e00070a0 2021-03-09T09:19:06.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:06.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:19:06.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:06.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d40 con 0x7f63e005c7c0 2021-03-09T09:19:06.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:06.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:06.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.335+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:06.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:06.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.336+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:06.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.336+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:06.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.336+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:19:06.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.336+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:06.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.336+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:06.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:06.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:06.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:06.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:06.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:06.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:06.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:06.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:06.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:19:06.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:19:06.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:06.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:06.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068da0 con 0x7f63e005c7c0 2021-03-09T09:19:06.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:06.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:06.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010db0 con 0x7f63e005c7c0 2021-03-09T09:19:06.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:19:06.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:06.351 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13490 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:06.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5031 cs=0 l=1 rev1=1 rx=0x7f63fc021ff0 tx=0x7f63fc04a7b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:06.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:19:06.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.337+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:06.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63e005c7c0 2021-03-09T09:19:06.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.338+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:06.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.338+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:06.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.338+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:06.353 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:06.354 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:06.354 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:06.354 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:06.354 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:06.354 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:06.355 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:06.355 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:06.355 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:06.355 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:06.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.339+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63e005c7c0 2021-03-09T09:19:06.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.339+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:06.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.339+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:19:06.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:19:06.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:06.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:06.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:07.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:07.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:07.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:07.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:08.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:08.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:08.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:08.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:09.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:09.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:09.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:09.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:09.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:09.695+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:10.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:10.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:10.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:11.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:11.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.337+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:11.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.337+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5031 cs=0 l=1 rev1=1 rx=0x7f63fc021ff0 tx=0x7f63fc04a7b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:11.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.337+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5031 cs=0 l=1 rev1=1 rx=0x7f63fc021ff0 tx=0x7f63fc04a7b0).stop 2021-03-09T09:19:11.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:11.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:11.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:11.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:11.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:19:11.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:11.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:11.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:11.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:11.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:11.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:11.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:11.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f63e00070a0 2021-03-09T09:19:11.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:11.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00096e0 con 0x7f6400145bd0 2021-03-09T09:19:11.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:11.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022e60 con 0x7f63e0064730 2021-03-09T09:19:11.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:11.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:11.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:11.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:11.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:11.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:11.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:19:11.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.338+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:11.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:11.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:11.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:11.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:19:11.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:19:11.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:11.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:11.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e110 con 0x7f63e0064730 2021-03-09T09:19:11.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:11.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:11.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00109f0 con 0x7f63e0064730 2021-03-09T09:19:11.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:19:11.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:11.357 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:11.339+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13530 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:11.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.340+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:19:11.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.340+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:11.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.340+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002e960 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4046440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:11.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f63e0064730 2021-03-09T09:19:11.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.342+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:11.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.342+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:11.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.342+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:11.361 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:11.361 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:11.362 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:11.362 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:11.362 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:11.363 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:11.364 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.342+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046830 con 0x7f63e0064730 2021-03-09T09:19:11.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.342+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.342+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:19:11.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0064730 2021-03-09T09:19:11.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:11.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:11.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:12.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:12.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:12.573+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:13.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:13.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:13.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:13.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:14.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:14.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:14.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:14.696+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:15.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:15.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:15.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:15.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:16.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:19:16.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:16.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002e960 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4046440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:16.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002e960 secure :-1 s=READY pgs=5048 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4046440).stop 2021-03-09T09:19:16.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:16.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:16.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002e960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:16.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002e960 unknown :-1 s=CLOSED pgs=5048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:16.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:19:16.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:16.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:16.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:16.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:16.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:16.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:16.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:16.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ab00 con 0x7f63e00070a0 2021-03-09T09:19:16.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:16.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f360 con 0x7f6400145bd0 2021-03-09T09:19:16.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:16.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:19:16.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.340+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:16.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.341+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:16.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.341+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:16.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.341+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:16.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.341+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:16.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:16.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:19:16.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:16.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:16.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:16.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:16.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:16.352 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:16.341+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13566 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:16.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:16.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:16.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:16.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:16.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:16.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:19:16.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:19:16.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:16.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:16.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c800 con 0x7f63e005c7c0 2021-03-09T09:19:16.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:16.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:16.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e110 con 0x7f63e005c7c0 2021-03-09T09:19:16.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:19:16.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.342+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:16.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.343+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:19:16.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.343+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:16.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.343+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5066 cs=0 l=1 rev1=1 rx=0x7f63f400d380 tx=0x7f63f400d700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:16.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034730 con 0x7f63e005c7c0 2021-03-09T09:19:16.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.344+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:16.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.344+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.344+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:16.360 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:16.361 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:16.362 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:16.362 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:16.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63e005c7c0 2021-03-09T09:19:16.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.346+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:16.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.346+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:19:16.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e005c7c0 2021-03-09T09:19:16.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:16.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:16.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:17.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:17.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:17.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:17.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:18.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:18.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:18.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:18.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:19.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:19.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:19.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:19.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:19.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:19.697+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:20.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:20.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:20.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:20.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:21.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:21.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:21.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5066 cs=0 l=1 rev1=1 rx=0x7f63f400d380 tx=0x7f63f400d700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:21.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5066 cs=0 l=1 rev1=1 rx=0x7f63f400d380 tx=0x7f63f400d700).stop 2021-03-09T09:19:21.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:21.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:21.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:21.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:21.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:19:21.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:21.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:21.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.342+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:21.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:21.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:21.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:21.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:21.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013440 con 0x7f63e00070a0 2021-03-09T09:19:21.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:21.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011db0 con 0x7f6400145bd0 2021-03-09T09:19:21.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:21.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067930 con 0x7f63e0064730 2021-03-09T09:19:21.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:21.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:21.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:21.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:21.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:21.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:21.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:19:21.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:21.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.343+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:21.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.344+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:21.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.344+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:21.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.344+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:21.354 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:21.344+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13596 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:21.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:21.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:21.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:21.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:21.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:21.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:19:21.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:19:21.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:21.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:21.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0064730 2021-03-09T09:19:21.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:21.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:21.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068c10 con 0x7f63e0064730 2021-03-09T09:19:21.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:19:21.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:21.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 secure :-1 s=READY pgs=5077 cs=0 l=1 rev1=1 rx=0x7f63e0025770 tx=0x7f63f001c300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:21.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:19:21.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.345+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:21.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0064730 2021-03-09T09:19:21.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.346+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:21.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.346+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:21.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.346+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:21.360 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:21.360 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:21.360 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:21.361 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:21.361 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:21.361 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:21.361 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:21.361 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:21.362 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:21.362 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:21.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000ed10 con 0x7f63e0064730 2021-03-09T09:19:21.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.348+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:21.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T09:19:21.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.348+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:19:21.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:21.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:21.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:22.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:22.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:22.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:22.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:22.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:22.574+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:23.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:23.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:23.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:23.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:24.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:24.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:24.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:24.698+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:25.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:25.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:25.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:25.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:26.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e005ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:19:26.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e005ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:26.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 secure :-1 s=READY pgs=5077 cs=0 l=1 rev1=1 rx=0x7f63e0025770 tx=0x7f63f001c300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:26.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 secure :-1 s=READY pgs=5077 cs=0 l=1 rev1=1 rx=0x7f63e0025770 tx=0x7f63f001c300).stop 2021-03-09T09:19:26.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:26.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:26.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e005ddf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:26.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 unknown :-1 s=CLOSED pgs=5077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:26.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:19:26.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:26.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:26.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:26.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:26.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:26.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:26.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:26.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e400 con 0x7f63e00070a0 2021-03-09T09:19:26.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:26.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d50 con 0x7f6400145bd0 2021-03-09T09:19:26.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:26.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d40 con 0x7f63e005c7c0 2021-03-09T09:19:26.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:26.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.344+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:26.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:26.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:26.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:26.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:26.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:19:26.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:26.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:26.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:26.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:26.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.345+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:26.356 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13642 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:26.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:26.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:26.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:26.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:26.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:26.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:19:26.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:19:26.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:26.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:26.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010db0 con 0x7f63e005c7c0 2021-03-09T09:19:26.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.346+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:26.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.347+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:26.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.347+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:19:26.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.347+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:19:26.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.347+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:26.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.347+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5091 cs=0 l=1 rev1=1 rx=0x7f63f0004ff0 tx=0x7f63f004e640).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:26.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.347+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:19:26.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.347+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:26.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e005c7c0 2021-03-09T09:19:26.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.348+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:26.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.348+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:26.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.348+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:26.363 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:26.363 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:26.363 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:26.363 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:26.363 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:26.364 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:26.364 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:26.364 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:26.364 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:26.365 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:26.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e005c7c0 2021-03-09T09:19:26.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.349+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:26.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.349+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:19:26.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.350+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004df40 con 0x7f63e005c7c0 2021-03-09T09:19:26.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:26.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:26.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:27.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:27.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:27.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:27.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:28.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:28.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:28.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:28.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:29.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:29.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:29.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:29.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:29.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:29.699+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:30.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:30.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:30.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:30.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:31.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:31.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:31.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5091 cs=0 l=1 rev1=1 rx=0x7f63f0004ff0 tx=0x7f63f004e640).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:31.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5091 cs=0 l=1 rev1=1 rx=0x7f63f0004ff0 tx=0x7f63f004e640).stop 2021-03-09T09:19:31.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:31.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:31.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:31.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:31.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:19:31.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:31.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:31.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:31.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:31.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:31.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:31.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:31.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:19:31.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:31.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00096e0 con 0x7f6400145bd0 2021-03-09T09:19:31.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:31.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011db0 con 0x7f63e0064730 2021-03-09T09:19:31.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:31.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:31.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:31.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:31.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.347+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:31.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.348+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:31.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.348+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:19:31.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.348+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:31.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.348+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:31.357 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:31.348+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13681 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:31.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.348+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:31.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.348+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:31.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.348+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:31.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:31.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:31.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:31.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:31.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:31.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:19:31.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:19:31.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:31.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:31.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.349+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d880 con 0x7f63e0064730 2021-03-09T09:19:31.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.350+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:31.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.350+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:31.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.350+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010db0 con 0x7f63e0064730 2021-03-09T09:19:31.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.350+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:19:31.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.350+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:31.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.350+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 rx=0x7f63f0000cf0 tx=0x7f63f0006bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:31.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.350+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:19:31.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.350+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:31.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0064730 2021-03-09T09:19:31.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.351+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:31.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.351+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:31.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.351+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:31.364 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:31.365 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:31.365 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:31.365 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:31.365 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:31.365 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:31.366 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:31.366 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:31.366 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:31.366 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:31.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0064730 2021-03-09T09:19:31.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.351+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:31.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.351+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:19:31.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63e0064730 2021-03-09T09:19:31.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:31.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:31.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:32.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:32.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:32.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:32.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:32.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:32.575+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:33.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:33.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:33.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:33.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:34.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:34.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:34.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:34.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:34.700+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:35.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:35.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e005ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e005ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 rx=0x7f63f0000cf0 tx=0x7f63f0006bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 secure :-1 s=READY pgs=5108 cs=0 l=1 rev1=1 rx=0x7f63f0000cf0 tx=0x7f63f0006bc0).stop 2021-03-09T09:19:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e005ddf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 unknown :-1 s=CLOSED pgs=5108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:19:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:36.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:36.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:36.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:36.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067930 con 0x7f63e00070a0 2021-03-09T09:19:36.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:36.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e400 con 0x7f6400145bd0 2021-03-09T09:19:36.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:36.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d40 con 0x7f63e005c7c0 2021-03-09T09:19:36.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:36.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:36.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:36.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:36.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.349+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:36.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.350+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:36.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.350+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:19:36.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.350+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:36.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.350+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:36.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.350+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:36.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.350+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:36.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.350+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:36.365 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:36.350+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13718 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:36.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:36.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:36.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:36.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:36.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:36.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:19:36.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:19:36.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:36.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:36.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002efe0 con 0x7f63e005c7c0 2021-03-09T09:19:36.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:36.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:36.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068da0 con 0x7f63e005c7c0 2021-03-09T09:19:36.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:19:36.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.351+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:36.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.352+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5120 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc04d5f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:36.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.352+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:19:36.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.352+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:36.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.353+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc006480 con 0x7f63e005c7c0 2021-03-09T09:19:36.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.353+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:36.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.353+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:36.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.353+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:36.371 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:36.372 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:36.372 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:36.372 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:36.372 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:36.372 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:36.373 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:36.373 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:36.373 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:36.373 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:36.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02bcb0 con 0x7f63e005c7c0 2021-03-09T09:19:36.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.355+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:36.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.355+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:19:36.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:19:36.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:36.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:36.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:37.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:37.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:37.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:37.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:38.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:38.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:38.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:38.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:39.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:39.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:39.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:39.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:39.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:39.701+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:40.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:40.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:40.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:40.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:41.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:19:41.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:41.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5120 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc04d5f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:41.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5120 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc04d5f0).stop 2021-03-09T09:19:41.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:41.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:41.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:41.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:41.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:19:41.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:41.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:41.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:41.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:41.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:41.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:41.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:41.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d50 con 0x7f63e00070a0 2021-03-09T09:19:41.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:41.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016030 con 0x7f6400145bd0 2021-03-09T09:19:41.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:41.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:19:41.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:41.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:41.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:41.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:41.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.351+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:41.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:41.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.352+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:19:41.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.352+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:41.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.352+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:41.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.352+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:41.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.352+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:41.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.352+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:41.362 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:41.352+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13757 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:41.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:41.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:41.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:41.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:41.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:41.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:19:41.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:19:41.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:41.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:41.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068c10 con 0x7f63e0064730 2021-03-09T09:19:41.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:41.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:41.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d880 con 0x7f63e0064730 2021-03-09T09:19:41.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:19:41.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.353+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:41.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.354+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:19:41.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.354+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:41.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.354+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 secure :-1 s=READY pgs=5132 cs=0 l=1 rev1=1 rx=0x7f63e0001a80 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:41.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0064730 2021-03-09T09:19:41.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.355+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:41.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.355+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:41.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.355+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:41.368 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:41.369 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:41.370 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:41.370 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:41.370 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:41.370 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:41.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0064730 2021-03-09T09:19:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.356+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.356+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:19:41.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.357+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T09:19:41.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:41.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:41.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:42.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:42.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:42.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:42.577+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:43.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:43.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:43.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:43.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:44.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:44.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:44.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:44.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:44.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:44.702+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:45.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:45.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:45.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:46.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e005ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:46.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e005ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:46.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 secure :-1 s=READY pgs=5132 cs=0 l=1 rev1=1 rx=0x7f63e0001a80 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:46.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 secure :-1 s=READY pgs=5132 cs=0 l=1 rev1=1 rx=0x7f63e0001a80 tx=0x7f63f000df50).stop 2021-03-09T09:19:46.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e005ddf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e005ddf0 unknown :-1 s=CLOSED pgs=5132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:19:46.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:46.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:46.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:46.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:46.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:46.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:46.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:46.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010690 con 0x7f63e00070a0 2021-03-09T09:19:46.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:46.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e320 con 0x7f6400145bd0 2021-03-09T09:19:46.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:46.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ad0 con 0x7f63e005c7c0 2021-03-09T09:19:46.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.353+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:46.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:46.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:46.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:46.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:19:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:46.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.354+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:46.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:46.364 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13794 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:46.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:46.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:46.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:46.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:19:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:19:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:46.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00109f0 con 0x7f63e005c7c0 2021-03-09T09:19:46.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:46.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:46.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002efe0 con 0x7f63e005c7c0 2021-03-09T09:19:46.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:19:46.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.355+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:46.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.356+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:19:46.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.356+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:46.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.356+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5147 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc03c590).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:46.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.357+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:19:46.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.357+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:46.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.357+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:46.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.357+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:46.371 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:46.371 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:46.371 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:46.371 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:46.372 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:46.372 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:46.372 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:46.372 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:46.372 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:46.373 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:46.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0715c0 con 0x7f63e005c7c0 2021-03-09T09:19:46.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.358+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:46.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.359+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:19:46.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.359+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:19:46.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:46.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:46.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:47.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:47.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:47.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:47.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:48.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:48.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:48.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:48.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:49.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:49.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:49.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:49.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:49.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:49.703+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:50.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:50.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:50.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:50.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:51.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:51.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:51.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5147 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc03c590).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:51.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5147 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc03c590).stop 2021-03-09T09:19:51.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:51.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:19:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:51.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:51.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:51.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:51.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d550 con 0x7f63e00070a0 2021-03-09T09:19:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f6400145bd0 2021-03-09T09:19:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:51.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e0064730 2021-03-09T09:19:51.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:51.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:51.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:51.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.356+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:19:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:51.366 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13832 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:51.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:19:51.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:19:51.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:51.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:51.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00018f0 con 0x7f63e0064730 2021-03-09T09:19:51.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:51.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:51.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.357+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c800 con 0x7f63e0064730 2021-03-09T09:19:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.358+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:19:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.358+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.358+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:19:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.358+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.358+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 secure :-1 s=READY pgs=5158 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f402af70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.359+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63e0064730 2021-03-09T09:19:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.359+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.359+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.359+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:51.376 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:51.376 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:51.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63e0064730 2021-03-09T09:19:51.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.360+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:51.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.360+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:19:51.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.361+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T09:19:51.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:51.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:51.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:52.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:52.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:52.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:52.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:52.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:52.577+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:53.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:53.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:53.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:53.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:54.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:54.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:54.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:54.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:54.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:54.704+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:19:55.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:55.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:55.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:55.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:56.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:19:56.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:19:56.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 secure :-1 s=READY pgs=5158 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f402af70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:19:56.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 secure :-1 s=READY pgs=5158 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f402af70).stop 2021-03-09T09:19:56.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:56.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:19:56.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:19:56.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 unknown :-1 s=CLOSED pgs=5158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:19:56.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:19:56.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:56.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:19:56.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:56.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:19:56.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:19:56.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:19:56.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:56.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d40 con 0x7f63e00070a0 2021-03-09T09:19:56.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:56.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:19:56.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:19:56.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067930 con 0x7f63e005c7c0 2021-03-09T09:19:56.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:19:56.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:19:56.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:56.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:19:56.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:19:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:19:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:19:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:19:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:19:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:19:56.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:19:56.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.358+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:19:56.369 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13870 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:19:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:19:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:19:56.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:19:56.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:19:56.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:19:56.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:19:56.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:56.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c8d0 con 0x7f63e005c7c0 2021-03-09T09:19:56.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:19:56.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:19:56.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00109f0 con 0x7f63e005c7c0 2021-03-09T09:19:56.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:19:56.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.359+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:56.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.361+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5177 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc075fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:19:56.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.361+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:19:56.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.361+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:19:56.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013f00 con 0x7f63e005c7c0 2021-03-09T09:19:56.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.362+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:19:56.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.362+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:19:56.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.362+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:19:56.376 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:19:56.376 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:19:56.376 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:56.376 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:19:56.376 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:19:56.377 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:19:56.377 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:19:56.377 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:19:56.377 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:19:56.377 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:19:56.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013f00 con 0x7f63e005c7c0 2021-03-09T09:19:56.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.365+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:19:56.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:19:56.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.365+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:19:56.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:56.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:56.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:57.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:57.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:57.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:57.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:58.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:58.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:58.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:58.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:59.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:59.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:19:59.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:59.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:19:59.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:19:59.705+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:00.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:00.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:00.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:00.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:01.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:20:01.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:01.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5177 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc075fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:01.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5177 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc075fe0).stop 2021-03-09T09:20:01.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:01.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:01.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:01.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:01.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:20:01.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:01.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:01.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:01.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:01.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:01.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:01.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:01.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e00070a0 2021-03-09T09:20:01.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:01.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b8a0 con 0x7f6400145bd0 2021-03-09T09:20:01.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:01.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e320 con 0x7f63e0064730 2021-03-09T09:20:01.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.360+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:01.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.361+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:01.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.361+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:01.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.361+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:01.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.361+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:01.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.361+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:01.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.361+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:20:01.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.361+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:01.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.361+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:01.379 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:01.361+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13906 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:01.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.362+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:01.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.362+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:01.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.362+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:01.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:01.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:01.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:01.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:01.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:01.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:20:01.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:20:01.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:01.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:01.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00133f0 con 0x7f63e0064730 2021-03-09T09:20:01.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:01.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:01.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00018f0 con 0x7f63e0064730 2021-03-09T09:20:01.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:20:01.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:01.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002f9d0 secure :-1 s=READY pgs=5188 cs=0 l=1 rev1=1 rx=0x7f63f40487b0 tx=0x7f63f4017f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:01.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:20:01.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.363+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:01.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63e0064730 2021-03-09T09:20:01.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.364+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:01.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.364+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:01.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.364+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:01.390 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:01.390 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:01.391 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:01.391 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:01.391 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:01.392 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:01.392 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:01.393 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:01.393 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:01.393 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:01.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63e0064730 2021-03-09T09:20:01.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.365+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:01.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.365+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:20:01.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.366+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e0064730 2021-03-09T09:20:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:01.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:02.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:02.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:02.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:02.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:02.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:02.579+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:03.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:03.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:03.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:03.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:04.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:04.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:04.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:04.706+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:05.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:05.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:05.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:05.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:06.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:06.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:06.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002f9d0 secure :-1 s=READY pgs=5188 cs=0 l=1 rev1=1 rx=0x7f63f40487b0 tx=0x7f63f4017f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:06.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002f9d0 secure :-1 s=READY pgs=5188 cs=0 l=1 rev1=1 rx=0x7f63f40487b0 tx=0x7f63f4017f40).stop 2021-03-09T09:20:06.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:06.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:06.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002f9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:06.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002f9d0 unknown :-1 s=CLOSED pgs=5188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:06.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:20:06.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:06.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:06.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:06.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:06.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:06.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:06.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:06.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:20:06.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:06.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f6400145bd0 2021-03-09T09:20:06.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:06.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067930 con 0x7f63e005c7c0 2021-03-09T09:20:06.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:06.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:06.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.362+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:06.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.363+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:06.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.363+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:06.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.363+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:06.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.363+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:20:06.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.363+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:06.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.363+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:06.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.363+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:06.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.363+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:06.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.363+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:06.373 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13946 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:06.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:06.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:06.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:06.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:06.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:06.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:20:06.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:20:06.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:06.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:06.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00109f0 con 0x7f63e005c7c0 2021-03-09T09:20:06.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:06.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:06.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00133f0 con 0x7f63e005c7c0 2021-03-09T09:20:06.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:20:06.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.364+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:06.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.365+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5198 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f404b350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:06.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.365+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:20:06.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.365+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:06.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.366+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009350 con 0x7f63e005c7c0 2021-03-09T09:20:06.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.366+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:06.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.366+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:06.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.366+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:06.379 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:06.379 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:06.379 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:06.380 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:06.380 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:06.380 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:06.380 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:06.380 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:06.381 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:06.381 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:06.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.367+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63e005c7c0 2021-03-09T09:20:06.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.367+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:06.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.367+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402d2a0 con 0x7f63e005c7c0 2021-03-09T09:20:06.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.368+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:20:06.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:06.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:07.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:07.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:07.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:07.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:08.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:08.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:09.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:09.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:09.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:09.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:09.707+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:10.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:10.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:10.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:10.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:11.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:20:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5198 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f404b350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5198 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f404b350).stop 2021-03-09T09:20:11.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:20:11.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:11.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e320 con 0x7f63e00070a0 2021-03-09T09:20:11.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:11.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cf0 con 0x7f6400145bd0 2021-03-09T09:20:11.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:11.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63e0064730 2021-03-09T09:20:11.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:11.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:11.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:11.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:11.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:11.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:11.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:20:11.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:11.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.365+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:11.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:11.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:11.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:11.376 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 13984 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:11.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:11.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:11.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:11.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:11.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:11.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:20:11.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:20:11.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:11.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:11.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017c70 con 0x7f63e0064730 2021-03-09T09:20:11.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:11.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:11.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00109f0 con 0x7f63e0064730 2021-03-09T09:20:11.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:20:11.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.366+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:11.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.367+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029c80 secure :-1 s=READY pgs=5211 cs=0 l=1 rev1=1 rx=0x7f63f402e010 tx=0x7f63f4003b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:11.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.367+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:20:11.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.367+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:11.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40367b0 con 0x7f63e0064730 2021-03-09T09:20:11.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.368+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:11.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.368+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:11.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.368+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:11.383 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:11.383 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:11.383 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:11.383 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:11.383 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:11.384 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:11.384 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:11.384 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:11.384 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:11.384 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:11.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400e260 con 0x7f63e0064730 2021-03-09T09:20:11.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.371+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:11.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.371+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:20:11.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63e0064730 2021-03-09T09:20:11.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:11.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:11.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:12.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:12.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:12.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:12.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:12.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:12.581+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:13.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:13.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:13.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:13.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:14.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:14.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:14.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:14.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:14.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:14.708+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:15.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:15.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:15.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:15.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:16.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.367+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0029c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:16.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.367+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0029c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:16.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.367+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029c80 secure :-1 s=READY pgs=5211 cs=0 l=1 rev1=1 rx=0x7f63f402e010 tx=0x7f63f4003b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:16.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.367+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029c80 secure :-1 s=READY pgs=5211 cs=0 l=1 rev1=1 rx=0x7f63f402e010 tx=0x7f63f4003b10).stop 2021-03-09T09:20:16.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.367+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:16.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:16.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0029c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:16.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0029c80 unknown :-1 s=CLOSED pgs=5211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:16.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:20:16.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:16.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:16.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:16.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:20:16.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a900 con 0x7f6400145bd0 2021-03-09T09:20:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023790 con 0x7f63e005c7c0 2021-03-09T09:20:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:16.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:16.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:20:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:16.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:16.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.368+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:16.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:16.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:16.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:16.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:16.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:20:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:20:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:16.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c700 con 0x7f63e005c7c0 2021-03-09T09:20:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068c10 con 0x7f63e005c7c0 2021-03-09T09:20:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:20:16.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:16.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5225 cs=0 l=1 rev1=1 rx=0x7f63e0037ed0 tx=0x7f63f0013b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:16.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:20:16.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:16.382 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:16.369+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14020 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:16.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.370+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005d2e0 con 0x7f63e005c7c0 2021-03-09T09:20:16.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.370+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:16.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.370+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:16.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.370+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:16.383 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:16.384 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:16.384 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:16.384 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:16.384 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:16.385 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:16.385 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:16.385 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:16.385 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:16.386 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:16.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.370+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001b040 con 0x7f63e005c7c0 2021-03-09T09:20:16.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.370+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:16.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.370+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:20:16.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038430 con 0x7f63e005c7c0 2021-03-09T09:20:16.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:16.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:16.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:17.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:17.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:17.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:17.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:18.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:18.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:18.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:18.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:19.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:19.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:19.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:19.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:19.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:19.709+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:20.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:20.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.370+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:20:21.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.370+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:21.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.370+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5225 cs=0 l=1 rev1=1 rx=0x7f63e0037ed0 tx=0x7f63f0013b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:21.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.370+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5225 cs=0 l=1 rev1=1 rx=0x7f63e0037ed0 tx=0x7f63f0013b10).stop 2021-03-09T09:20:21.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:21.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:21.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:21.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:21.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:20:21.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:21.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:21.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:21.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:21.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067930 con 0x7f63e00070a0 2021-03-09T09:20:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018970 con 0x7f6400145bd0 2021-03-09T09:20:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:21.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63e0064730 2021-03-09T09:20:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:21.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:21.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.371+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:21.381 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:21.372+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14060 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:21.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.372+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.372+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:20:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.372+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.372+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:21.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.372+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:21.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.372+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:21.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.372+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:21.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:21.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:21.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:21.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:20:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:20:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:21.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:21.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010db0 con 0x7f63e0064730 2021-03-09T09:20:21.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:21.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:21.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017c70 con 0x7f63e0064730 2021-03-09T09:20:21.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:20:21.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:21.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:20:21.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:21.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 secure :-1 s=READY pgs=5241 cs=0 l=1 rev1=1 rx=0x7f63f404cf50 tx=0x7f63f4005180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:21.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404b780 con 0x7f63e0064730 2021-03-09T09:20:21.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:21.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:21.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.373+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:21.389 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:21.390 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:21.390 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:21.390 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:21.390 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:21.391 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:21.391 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:21.391 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:21.391 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:21.391 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:21.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63e0064730 2021-03-09T09:20:21.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.374+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:21.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.374+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:20:21.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e0064730 2021-03-09T09:20:21.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:21.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:21.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:22.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:22.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:22.582+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:23.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:23.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:23.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:23.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:24.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:24.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:24.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:24.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:24.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:24.710+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:25.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:25.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:25.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:25.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:26.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:26.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:26.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 secure :-1 s=READY pgs=5241 cs=0 l=1 rev1=1 rx=0x7f63f404cf50 tx=0x7f63f4005180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:26.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 secure :-1 s=READY pgs=5241 cs=0 l=1 rev1=1 rx=0x7f63f404cf50 tx=0x7f63f4005180).stop 2021-03-09T09:20:26.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:26.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:26.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:26.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000af40 unknown :-1 s=CLOSED pgs=5241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:26.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:20:26.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:26.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:26.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:26.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:26.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:26.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:26.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:26.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e320 con 0x7f63e00070a0 2021-03-09T09:20:26.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:26.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:20:26.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:26.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a380 con 0x7f63e005c7c0 2021-03-09T09:20:26.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.372+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:26.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.373+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:26.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.373+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:26.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.373+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:26.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.373+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:26.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.373+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:26.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.373+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:20:26.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.373+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:26.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.373+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:26.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.374+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:26.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.374+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:26.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.374+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:26.390 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:26.374+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14098 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:26.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.374+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:26.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.374+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:26.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.374+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:26.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.374+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:26.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.374+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:26.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:20:26.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:20:26.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:26.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:26.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001030 con 0x7f63e005c7c0 2021-03-09T09:20:26.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:26.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:26.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010db0 con 0x7f63e005c7c0 2021-03-09T09:20:26.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:20:26.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:26.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:20:26.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:26.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.375+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5257 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404cc70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:26.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f63e005c7c0 2021-03-09T09:20:26.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.377+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:26.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.377+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:26.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.377+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:26.406 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:26.407 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:26.408 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:26.409 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:26.410 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:26.411 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:26.411 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:26.412 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:26.412 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:26.413 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:26.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63e005c7c0 2021-03-09T09:20:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.378+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.378+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:20:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4037040 con 0x7f63e005c7c0 2021-03-09T09:20:26.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:26.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:26.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:27.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:27.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:27.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:27.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:28.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:28.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:28.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:28.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:29.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:29.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:29.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:29.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:29.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:29.711+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:30.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:30.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:30.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:30.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:31.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:31.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:31.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5257 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404cc70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:31.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5257 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404cc70).stop 2021-03-09T09:20:31.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:31.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:31.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:31.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:31.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:20:31.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:31.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:31.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:31.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:31.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:31.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:31.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:31.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065b90 con 0x7f63e00070a0 2021-03-09T09:20:31.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:31.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:31.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018770 con 0x7f6400145bd0 2021-03-09T09:20:31.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:31.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e400 con 0x7f63e0064730 2021-03-09T09:20:31.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:31.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.374+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:31.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.375+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:31.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.375+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:31.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.375+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:31.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.375+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:20:31.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.375+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:31.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.375+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:31.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.376+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:31.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.376+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:31.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.376+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:31.385 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:31.376+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14136 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:31.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:31.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:31.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:31.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:31.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:31.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:20:31.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:20:31.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:31.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:31.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010db0 con 0x7f63e0064730 2021-03-09T09:20:31.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:31.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:31.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c700 con 0x7f63e0064730 2021-03-09T09:20:31.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:20:31.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:31.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:20:31.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 secure :-1 s=READY pgs=5271 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:31.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.377+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:31.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00542c0 con 0x7f63e0064730 2021-03-09T09:20:31.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.379+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:31.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.379+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:31.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.379+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:31.392 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:31.392 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:31.393 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:31.393 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:31.393 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:31.393 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:31.393 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:31.394 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:31.394 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:31.394 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:31.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.381+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T09:20:31.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.381+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:31.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.381+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:20:31.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.382+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001b040 con 0x7f63e0064730 2021-03-09T09:20:31.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:31.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:31.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:32.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:32.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:32.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:32.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:32.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:32.583+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:33.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:33.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:33.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:33.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:34.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:34.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:34.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:34.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:34.712+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:35.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:35.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:35.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:35.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:36.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00686c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:36.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00686c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:36.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 secure :-1 s=READY pgs=5271 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:36.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 secure :-1 s=READY pgs=5271 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f004d180).stop 2021-03-09T09:20:36.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:36.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:36.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00686c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:36.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 unknown :-1 s=CLOSED pgs=5271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:36.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:20:36.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:36.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:36.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:36.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:36.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:36.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:36.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:36.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018970 con 0x7f63e00070a0 2021-03-09T09:20:36.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:36.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:20:36.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:36.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f030 con 0x7f63e005c7c0 2021-03-09T09:20:36.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:36.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:36.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:36.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:36.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:36.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:36.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:20:36.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:36.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.377+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:36.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.378+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:36.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.378+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:36.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.378+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:36.387 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14174 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:36.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:36.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:36.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:36.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:36.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:36.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:20:36.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:20:36.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:36.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:36.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026f70 con 0x7f63e005c7c0 2021-03-09T09:20:36.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:36.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:36.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001030 con 0x7f63e005c7c0 2021-03-09T09:20:36.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:20:36.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.379+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:36.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.380+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5282 cs=0 l=1 rev1=1 rx=0x7f63f40036c0 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:36.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.380+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:20:36.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.380+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:36.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.382+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40462c0 con 0x7f63e005c7c0 2021-03-09T09:20:36.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.382+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:36.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.382+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:36.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.382+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:36.393 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:36.393 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:36.393 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:36.393 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:36.393 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:36.394 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:36.394 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:36.394 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:36.394 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:36.394 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:36.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.382+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405c2f0 con 0x7f63e005c7c0 2021-03-09T09:20:36.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.382+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:36.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.382+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:20:36.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405c2f0 con 0x7f63e005c7c0 2021-03-09T09:20:36.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:36.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:36.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:37.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:37.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:37.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:37.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:38.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:38.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:38.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:38.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:39.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:39.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:39.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:39.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:39.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:39.712+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:40.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:40.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:40.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:40.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:41.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:41.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:41.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5282 cs=0 l=1 rev1=1 rx=0x7f63f40036c0 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:41.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5282 cs=0 l=1 rev1=1 rx=0x7f63f40036c0 tx=0x7f63f4017ed0).stop 2021-03-09T09:20:41.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:41.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:41.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:41.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:41.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:20:41.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:41.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:41.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:41.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:41.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:41.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:41.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:41.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015380 con 0x7f63e00070a0 2021-03-09T09:20:41.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:41.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011d70 con 0x7f6400145bd0 2021-03-09T09:20:41.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:41.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077f10 con 0x7f63e0064730 2021-03-09T09:20:41.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:41.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:41.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.378+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:41.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.379+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:41.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.379+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:41.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.379+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:41.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.379+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:20:41.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.379+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:41.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.379+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:41.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.379+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:41.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.379+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:41.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.379+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:41.389 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14209 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:41.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:41.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:41.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:41.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:41.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:41.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:20:41.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:20:41.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:41.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:41.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012790 con 0x7f63e0064730 2021-03-09T09:20:41.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:41.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:41.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c8d0 con 0x7f63e0064730 2021-03-09T09:20:41.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:20:41.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.380+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:41.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.381+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:20:41.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.381+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:41.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.381+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 secure :-1 s=READY pgs=5300 cs=0 l=1 rev1=1 rx=0x7f63e0030610 tx=0x7f63fc009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:41.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc006480 con 0x7f63e0064730 2021-03-09T09:20:41.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.383+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:41.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.383+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:41.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.383+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:41.395 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:41.396 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:41.396 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:41.396 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:41.397 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:41.397 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:41.397 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:41.397 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:41.397 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:41.398 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:41.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0715c0 con 0x7f63e0064730 2021-03-09T09:20:41.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.385+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:41.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.385+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:20:41.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0064730 2021-03-09T09:20:41.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:41.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:41.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:42.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:42.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:42.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:42.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:42.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:42.584+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:43.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:43.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:43.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:43.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:44.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:44.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:44.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:44.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:44.713+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:45.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:45.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:45.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:45.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:46.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:46.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:46.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 secure :-1 s=READY pgs=5300 cs=0 l=1 rev1=1 rx=0x7f63e0030610 tx=0x7f63fc009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:46.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 secure :-1 s=READY pgs=5300 cs=0 l=1 rev1=1 rx=0x7f63e0030610 tx=0x7f63fc009a70).stop 2021-03-09T09:20:46.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:46.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:46.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:46.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 unknown :-1 s=CLOSED pgs=5300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:46.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:20:46.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:46.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:46.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:46.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:46.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:46.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:46.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:46.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017640 con 0x7f63e00070a0 2021-03-09T09:20:46.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:46.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00155c0 con 0x7f6400145bd0 2021-03-09T09:20:46.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:46.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013640 con 0x7f63e005c7c0 2021-03-09T09:20:46.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.380+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:46.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.381+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:46.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.381+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:46.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.381+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:46.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.381+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:46.389 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:46.382+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14257 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:46.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.382+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:46.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.382+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:20:46.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.382+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:46.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.382+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:46.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.382+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:46.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.382+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:46.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.382+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:46.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:46.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:46.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:20:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:20:46.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:46.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:46.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067330 con 0x7f63e005c7c0 2021-03-09T09:20:46.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:46.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:46.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026f70 con 0x7f63e005c7c0 2021-03-09T09:20:46.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:20:46.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.383+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:46.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.384+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5314 cs=0 l=1 rev1=1 rx=0x7f63f405bc10 tx=0x7f63f404cd50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:46.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.384+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:20:46.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.384+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:46.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405beb0 con 0x7f63e005c7c0 2021-03-09T09:20:46.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.386+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:46.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.386+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:46.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.386+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:46.397 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:46.397 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:46.397 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:46.397 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:46.397 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:46.398 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:46.398 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:46.398 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:46.399 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:46.399 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:46.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034dd0 con 0x7f63e005c7c0 2021-03-09T09:20:46.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.386+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:46.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63e005c7c0 2021-03-09T09:20:46.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.387+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:20:46.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:46.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:46.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:47.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:47.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:47.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:47.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:48.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:48.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:48.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:48.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:49.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:49.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:49.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:49.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:49.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:49.714+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:50.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:50.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:50.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:50.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:51.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:20:51.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:51.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5314 cs=0 l=1 rev1=1 rx=0x7f63f405bc10 tx=0x7f63f404cd50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:51.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5314 cs=0 l=1 rev1=1 rx=0x7f63f405bc10 tx=0x7f63f404cd50).stop 2021-03-09T09:20:51.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:51.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:51.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:51.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:51.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:20:51.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:51.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:51.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:51.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:51.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:51.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:51.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:51.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:20:51.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:51.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077f10 con 0x7f6400145bd0 2021-03-09T09:20:51.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:51.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f5f0 con 0x7f63e0064730 2021-03-09T09:20:51.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:51.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:51.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:51.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:51.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.383+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:51.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.384+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:51.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.384+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:20:51.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.384+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:51.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.384+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:51.393 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:51.385+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14295 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:51.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.385+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:51.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.385+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:51.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.385+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:51.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:20:51.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:20:51.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:51.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:51.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d60 con 0x7f63e0064730 2021-03-09T09:20:51.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067330 con 0x7f63e0064730 2021-03-09T09:20:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:20:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:51.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 secure :-1 s=READY pgs=5328 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4061e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:51.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:20:51.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.386+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:51.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034c40 con 0x7f63e0064730 2021-03-09T09:20:51.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.387+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:51.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.387+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:51.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.387+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:51.401 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:51.401 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:51.403 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034870 con 0x7f63e0064730 2021-03-09T09:20:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.388+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.388+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:20:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.390+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405bd30 con 0x7f63e0064730 2021-03-09T09:20:51.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:51.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:51.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:52.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:52.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:52.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:52.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:52.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:52.585+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:53.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:53.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:53.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:53.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:54.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:54.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:54.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:54.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:54.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:54.715+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:20:55.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:55.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:55.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:55.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:56.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:20:56.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:20:56.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 secure :-1 s=READY pgs=5328 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4061e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:20:56.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 secure :-1 s=READY pgs=5328 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4061e30).stop 2021-03-09T09:20:56.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:56.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:20:56.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:20:56.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0001030 unknown :-1 s=CLOSED pgs=5328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:20:56.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:20:56.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:56.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:20:56.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:56.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:20:56.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:20:56.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:20:56.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:56.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e400 con 0x7f63e00070a0 2021-03-09T09:20:56.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:56.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e520 con 0x7f6400145bd0 2021-03-09T09:20:56.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:20:56.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00155c0 con 0x7f63e005c7c0 2021-03-09T09:20:56.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:20:56.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.385+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:20:56.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:56.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:20:56.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:20:56.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:20:56.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:20:56.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:20:56.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:20:56.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:20:56.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:20:56.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.386+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:20:56.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:20:56.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:56.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:56.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:20:56.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:20:56.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:20:56.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:20:56.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:20:56.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:56.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029890 con 0x7f63e005c7c0 2021-03-09T09:20:56.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:20:56.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:20:56.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012790 con 0x7f63e005c7c0 2021-03-09T09:20:56.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:20:56.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:56.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.388+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5339 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc03f280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:20:56.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.388+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:20:56.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.388+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:20:56.401 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:20:56.387+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14333 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:20:56.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63e005c7c0 2021-03-09T09:20:56.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.388+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:20:56.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.388+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:20:56.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.388+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:20:56.403 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:20:56.403 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:20:56.403 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:56.403 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:20:56.403 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:20:56.404 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:20:56.404 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:20:56.404 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:20:56.404 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:20:56.404 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:20:56.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.389+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:20:56.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.389+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:20:56.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.390+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:20:56.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.391+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:20:56.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:56.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:56.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:57.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:57.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:57.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:57.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:58.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:58.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:58.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:58.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:59.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:59.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:20:59.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:59.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:20:59.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:20:59.716+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:00.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:00.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:00.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:00.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:01.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:21:01.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:01.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5339 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc03f280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:01.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5339 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc03f280).stop 2021-03-09T09:21:01.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:01.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:01.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:01.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:01.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:21:01.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:01.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:01.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:01.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:01.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:01.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:01.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:01.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:21:01.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:01.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e400 con 0x7f6400145bd0 2021-03-09T09:21:01.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:01.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0064730 2021-03-09T09:21:01.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:01.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:01.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.388+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:01.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.389+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:01.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.389+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:01.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.389+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:01.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.389+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:21:01.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.389+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:01.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.389+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:01.398 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:01.390+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14371 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:01.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.390+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:01.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.390+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:01.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.390+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:01.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:01.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:01.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:01.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:01.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:01.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:21:01.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:21:01.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:01.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:01.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bab0 con 0x7f63e0064730 2021-03-09T09:21:01.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:01.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:01.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010db0 con 0x7f63e0064730 2021-03-09T09:21:01.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:21:01.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.391+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:01.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.393+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:21:01.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.393+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 secure :-1 s=READY pgs=5357 cs=0 l=1 rev1=1 rx=0x7f63f0048be0 tx=0x7f63f0024f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:01.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.393+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:01.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0064730 2021-03-09T09:21:01.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.393+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:01.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.393+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:01.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.393+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:01.405 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:01.406 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:01.406 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:01.406 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:01.406 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:01.406 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:01.407 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:01.407 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:01.407 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:01.407 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:01.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0064730 2021-03-09T09:21:01.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.396+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:01.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.396+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:21:01.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001b040 con 0x7f63e0064730 2021-03-09T09:21:01.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:01.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:01.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:02.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:02.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:02.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:02.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:02.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:02.586+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:03.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:03.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:04.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:04.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:04.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:04.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:04.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:04.717+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:05.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:05.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:05.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:05.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:06.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00686c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:21:06.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00686c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:06.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 secure :-1 s=READY pgs=5357 cs=0 l=1 rev1=1 rx=0x7f63f0048be0 tx=0x7f63f0024f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:06.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 secure :-1 s=READY pgs=5357 cs=0 l=1 rev1=1 rx=0x7f63f0048be0 tx=0x7f63f0024f80).stop 2021-03-09T09:21:06.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:06.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:06.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00686c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:06.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00686c0 unknown :-1 s=CLOSED pgs=5357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:06.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:21:06.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:06.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:06.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:06.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:06.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002aab0 con 0x7f63e00070a0 2021-03-09T09:21:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e520 con 0x7f6400145bd0 2021-03-09T09:21:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00198c0 con 0x7f63e005c7c0 2021-03-09T09:21:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.390+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.391+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.391+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.391+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.391+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:21:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.391+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.391+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.391+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:06.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.391+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:06.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.392+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:06.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.392+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:06.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.392+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:06.402 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:06.392+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14409 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:06.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:06.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:06.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:06.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:06.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:06.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:21:06.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:21:06.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:06.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:06.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001e10 con 0x7f63e005c7c0 2021-03-09T09:21:06.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:06.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:06.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029890 con 0x7f63e005c7c0 2021-03-09T09:21:06.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:21:06.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.393+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:06.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.394+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5368 cs=0 l=1 rev1=1 rx=0x7f63fc04d630 tx=0x7f63fc06f370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:06.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.394+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:21:06.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.394+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:06.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:21:06.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.396+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:06.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.396+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:06.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.396+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:06.408 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:06.408 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:06.408 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:06.408 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:06.409 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:06.409 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:06.409 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:06.409 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:06.409 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:06.410 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:06.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.398+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0715c0 con 0x7f63e005c7c0 2021-03-09T09:21:06.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.398+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:06.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.398+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:21:06.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.398+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:21:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:06.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:07.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:07.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:07.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:07.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:08.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:08.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:08.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:08.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:09.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:09.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:09.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:09.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:09.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:09.718+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:10.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:10.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:10.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:10.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:11.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:21:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5368 cs=0 l=1 rev1=1 rx=0x7f63fc04d630 tx=0x7f63fc06f370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5368 cs=0 l=1 rev1=1 rx=0x7f63fc04d630 tx=0x7f63fc06f370).stop 2021-03-09T09:21:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:21:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:11.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:11.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:11.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:21:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018790 con 0x7f6400145bd0 2021-03-09T09:21:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bd70 con 0x7f63e0064730 2021-03-09T09:21:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:11.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:11.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.393+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:11.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.394+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:11.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.394+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:11.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.394+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:11.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.394+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:21:11.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.394+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:11.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.394+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:11.404 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:11.395+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14447 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:11.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.395+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:11.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.395+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:11.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.395+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:11.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:11.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:11.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:11.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:11.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:11.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:21:11.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:21:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cb80 con 0x7f63e0064730 2021-03-09T09:21:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:11.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bab0 con 0x7f63e0064730 2021-03-09T09:21:11.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:21:11.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.396+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:11.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.397+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f290 secure :-1 s=READY pgs=5385 cs=0 l=1 rev1=1 rx=0x7f63f000ee80 tx=0x7f63f000eef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.397+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:21:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.397+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000ecc0 con 0x7f63e0064730 2021-03-09T09:21:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.399+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.399+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:11.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.399+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:11.411 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:11.411 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:11.411 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:11.411 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:11.413 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.401+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000ecc0 con 0x7f63e0064730 2021-03-09T09:21:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.401+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.402+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:21:11.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.402+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001b040 con 0x7f63e0064730 2021-03-09T09:21:11.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:11.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:11.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:12.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:12.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:12.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:12.587+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:13.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:13.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:13.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:13.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:14.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:14.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:14.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:14.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:14.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:14.719+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:15.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:15.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:15.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:15.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:16.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.395+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:16.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.395+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.395+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f290 secure :-1 s=READY pgs=5385 cs=0 l=1 rev1=1 rx=0x7f63f000ee80 tx=0x7f63f000eef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.395+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f290 secure :-1 s=READY pgs=5385 cs=0 l=1 rev1=1 rx=0x7f63f000ee80 tx=0x7f63f000eef0).stop 2021-03-09T09:21:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000f290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000f290 unknown :-1 s=CLOSED pgs=5385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:21:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:16.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002aab0 con 0x7f63e00070a0 2021-03-09T09:21:16.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:16.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:21:16.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:16.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ef0 con 0x7f63e005c7c0 2021-03-09T09:21:16.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:16.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:16.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:16.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:16.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.396+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:16.405 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:16.397+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14485 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:16.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.397+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:16.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.397+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:21:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.397+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.397+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.397+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.397+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.397+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:16.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:16.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:16.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:16.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:16.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:16.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:21:16.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:21:16.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:16.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:16.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e440 con 0x7f63e005c7c0 2021-03-09T09:21:16.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:16.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002d60 con 0x7f63e005c7c0 2021-03-09T09:21:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:21:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.398+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.399+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5395 cs=0 l=1 rev1=1 rx=0x7f63f400d700 tx=0x7f63f4017f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.399+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:21:16.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.399+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:16.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e005c7c0 2021-03-09T09:21:16.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.400+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:16.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.400+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:16.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.400+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:16.413 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:16.413 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:16.413 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:16.414 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:16.414 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:16.414 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:16.415 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:16.415 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:16.415 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:16.415 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:16.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.401+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e005c7c0 2021-03-09T09:21:16.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.401+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:16.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.401+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:21:16.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.402+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63e005c7c0 2021-03-09T09:21:16.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:16.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:16.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:17.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:17.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:17.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:17.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:18.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:18.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:18.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:18.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:19.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:19.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:19.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:19.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:19.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:19.720+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:20.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:20.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:20.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:20.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:21.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:21.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:21.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5395 cs=0 l=1 rev1=1 rx=0x7f63f400d700 tx=0x7f63f4017f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:21.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5395 cs=0 l=1 rev1=1 rx=0x7f63f400d700 tx=0x7f63f4017f40).stop 2021-03-09T09:21:21.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:21.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:21.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:21.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:21.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:21:21.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:21.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:21.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:21.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:21.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:21.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:21.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:21.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d470 con 0x7f63e00070a0 2021-03-09T09:21:21.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:21.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:21:21.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:21.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d6b0 con 0x7f63e0064730 2021-03-09T09:21:21.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.397+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:21.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.398+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:21.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.398+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.398+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.398+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.398+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:21:21.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:21.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:21.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:21.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:21.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:21.409 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14523 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:21.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:21.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:21.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:21:21.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:21:21.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a700 con 0x7f63e0064730 2021-03-09T09:21:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.399+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cb80 con 0x7f63e0064730 2021-03-09T09:21:21.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:21:21.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:21.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000fab0 secure :-1 s=READY pgs=5404 cs=0 l=1 rev1=1 rx=0x7f63f0055e20 tx=0x7f63f005ee00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:21:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055c60 con 0x7f63e0064730 2021-03-09T09:21:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:21.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:21.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:21.416 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:21.416 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:21.417 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:21.417 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:21.417 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:21.417 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:21.417 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:21.418 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:21.418 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:21.418 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:21.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0071030 con 0x7f63e0064730 2021-03-09T09:21:21.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:21.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.400+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:21:21.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.402+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f000ec50 con 0x7f63e0064730 2021-03-09T09:21:21.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:21.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:21.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:22.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:22.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:22.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:22.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:22.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:22.589+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:23.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:23.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:23.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:23.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:24.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:24.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:24.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:24.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:24.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:24.721+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:25.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:25.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:25.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:25.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:26.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:21:26.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:26.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000fab0 secure :-1 s=READY pgs=5404 cs=0 l=1 rev1=1 rx=0x7f63f0055e20 tx=0x7f63f005ee00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:26.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000fab0 secure :-1 s=READY pgs=5404 cs=0 l=1 rev1=1 rx=0x7f63f0055e20 tx=0x7f63f005ee00).stop 2021-03-09T09:21:26.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:26.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:26.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000fab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:26.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000fab0 unknown :-1 s=CLOSED pgs=5404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:26.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:21:26.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:26.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:26.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:26.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:26.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:26.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:26.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:26.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023ff0 con 0x7f63e00070a0 2021-03-09T09:21:26.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:26.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018790 con 0x7f6400145bd0 2021-03-09T09:21:26.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:26.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e005c7c0 2021-03-09T09:21:26.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.399+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:26.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.400+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:26.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.400+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:26.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.400+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:26.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.400+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:26.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.401+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:26.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.401+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:21:26.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.401+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:26.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.401+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:26.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.401+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:26.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.401+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:26.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.401+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:26.411 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:26.401+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14559 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:26.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:26.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:26.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:26.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:26.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:21:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:21:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014780 con 0x7f63e005c7c0 2021-03-09T09:21:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e440 con 0x7f63e005c7c0 2021-03-09T09:21:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:21:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.402+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.403+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5422 cs=0 l=1 rev1=1 rx=0x7f63f400fce0 tx=0x7f63f400ff50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.403+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:21:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.403+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63e005c7c0 2021-03-09T09:21:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.404+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.404+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.404+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:26.418 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:26.418 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:26.418 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:26.418 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:26.418 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:26.419 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63e005c7c0 2021-03-09T09:21:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.405+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.405+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:21:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.406+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63e005c7c0 2021-03-09T09:21:26.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:26.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:26.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:27.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:27.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:27.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:27.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:28.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:28.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:28.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:28.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:29.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:29.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:29.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:29.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:29.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:29.722+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:30.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:30.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:30.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:30.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:31.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:21:31.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:31.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5422 cs=0 l=1 rev1=1 rx=0x7f63f400fce0 tx=0x7f63f400ff50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:31.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5422 cs=0 l=1 rev1=1 rx=0x7f63f400fce0 tx=0x7f63f400ff50).stop 2021-03-09T09:21:31.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:31.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:31.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:31.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:31.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:21:31.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:31.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:31.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:31.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:31.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:31.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:31.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:31.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077f10 con 0x7f63e00070a0 2021-03-09T09:21:31.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:31.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ef0 con 0x7f6400145bd0 2021-03-09T09:21:31.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:31.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002aab0 con 0x7f63e0064730 2021-03-09T09:21:31.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:31.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:31.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:31.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:31.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:31.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:31.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:21:31.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:31.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.402+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:31.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.403+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:31.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.403+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:31.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.403+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:31.413 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:31.403+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14599 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:31.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:31.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:31.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:31.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:31.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:31.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:21:31.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:21:31.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:31.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:31.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010db0 con 0x7f63e0064730 2021-03-09T09:21:31.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:31.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:31.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001e10 con 0x7f63e0064730 2021-03-09T09:21:31.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:21:31.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:31.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008d20 secure :-1 s=READY pgs=5436 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04d5f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:31.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:21:31.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.405+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:31.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.406+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071690 con 0x7f63e0064730 2021-03-09T09:21:31.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.406+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:31.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.406+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:31.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.406+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:31.419 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:31.419 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:31.419 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:31.420 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:31.420 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:31.420 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:31.420 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:31.420 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:31.421 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:31.421 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:31.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.407+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071690 con 0x7f63e0064730 2021-03-09T09:21:31.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.407+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:31.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.407+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:21:31.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e0064730 2021-03-09T09:21:31.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:31.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:31.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:32.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:32.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:32.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:32.590+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:33.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:33.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:33.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:33.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:34.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:34.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:34.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:34.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:34.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:34.723+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:35.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:35.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:35.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:35.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:36.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:36.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0008d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:36.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008d20 secure :-1 s=READY pgs=5436 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04d5f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:36.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008d20 secure :-1 s=READY pgs=5436 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04d5f0).stop 2021-03-09T09:21:36.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:36.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:36.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0008d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:36.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0008d20 unknown :-1 s=CLOSED pgs=5436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:36.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:21:36.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:36.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:36.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:36.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:36.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:36.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:36.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:36.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:21:36.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:36.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023790 con 0x7f6400145bd0 2021-03-09T09:21:36.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:36.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002aab0 con 0x7f63e005c7c0 2021-03-09T09:21:36.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:36.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:36.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.404+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:36.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.405+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:36.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.405+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:36.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.405+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:36.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.405+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:21:36.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.405+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:36.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.405+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:36.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:36.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:36.415 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14637 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:36.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:36.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:36.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:36.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:36.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:36.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:36.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:21:36.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:21:36.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:36.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:36.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d60 con 0x7f63e005c7c0 2021-03-09T09:21:36.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:36.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:36.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010db0 con 0x7f63e005c7c0 2021-03-09T09:21:36.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:21:36.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.406+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:36.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.407+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5450 cs=0 l=1 rev1=1 rx=0x7f63fc003ad0 tx=0x7f63fc066fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:36.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.407+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:21:36.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.407+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:36.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.408+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065450 con 0x7f63e005c7c0 2021-03-09T09:21:36.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.408+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:36.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.408+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:36.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.408+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:36.422 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:36.422 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:36.422 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:36.422 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:36.422 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:36.423 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:36.423 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:36.423 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:36.423 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:36.424 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:36.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032520 con 0x7f63e005c7c0 2021-03-09T09:21:36.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.409+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:36.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.409+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:21:36.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.411+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f63e005c7c0 2021-03-09T09:21:36.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:36.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:36.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:37.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:37.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:37.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:37.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:38.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:38.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:38.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:38.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:39.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:39.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:39.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:39.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:39.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:39.724+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:40.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:40.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:40.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:40.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:41.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.406+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:41.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.406+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:41.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.406+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5450 cs=0 l=1 rev1=1 rx=0x7f63fc003ad0 tx=0x7f63fc066fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:41.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.406+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 secure :-1 s=READY pgs=5450 cs=0 l=1 rev1=1 rx=0x7f63fc003ad0 tx=0x7f63fc066fa0).stop 2021-03-09T09:21:41.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:41.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:41.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c6c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:41.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c6c0 unknown :-1 s=CLOSED pgs=5450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:41.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:21:41.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:41.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:41.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:41.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:41.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:41.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:41.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:41.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e710 con 0x7f63e00070a0 2021-03-09T09:21:41.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:41.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016dd0 con 0x7f6400145bd0 2021-03-09T09:21:41.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:41.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:21:41.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:41.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:41.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:41.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:41.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.407+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:41.416 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:41.408+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14675 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:41.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:41.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:21:41.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:41.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:41.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:41.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:41.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:41.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:41.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:41.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:41.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:41.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:41.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:21:41.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:21:41.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:41.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:41.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017c30 con 0x7f63e0064730 2021-03-09T09:21:41.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:41.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:41.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a700 con 0x7f63e0064730 2021-03-09T09:21:41.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:21:41.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:41.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=5468 cs=0 l=1 rev1=1 rx=0x7f63f0000bc0 tx=0x7f63f0048840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:41.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:21:41.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.410+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:41.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.411+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0064730 2021-03-09T09:21:41.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.411+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:41.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.411+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:41.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.411+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:41.425 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:41.425 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:41.426 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:41.426 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:41.426 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:41.426 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:41.426 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:41.427 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:41.427 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:41.427 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:41.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0064730 2021-03-09T09:21:41.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.412+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:41.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.412+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:21:41.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001b040 con 0x7f63e0064730 2021-03-09T09:21:41.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:41.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:41.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:42.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:42.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:42.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:42.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:42.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:42.592+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:43.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:43.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:43.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:43.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:44.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:44.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:44.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:44.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:44.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:44.725+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:45.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:45.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:45.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:46.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:46.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:46.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=5468 cs=0 l=1 rev1=1 rx=0x7f63f0000bc0 tx=0x7f63f0048840).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:46.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=5468 cs=0 l=1 rev1=1 rx=0x7f63f0000bc0 tx=0x7f63f0048840).stop 2021-03-09T09:21:46.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:46.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:46.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:46.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=CLOSED pgs=5468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:46.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:21:46.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:46.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:46.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:46.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:46.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:46.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:46.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:46.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026f40 con 0x7f63e00070a0 2021-03-09T09:21:46.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:46.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014640 con 0x7f6400145bd0 2021-03-09T09:21:46.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:46.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023790 con 0x7f63e005c7c0 2021-03-09T09:21:46.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:46.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:46.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:46.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:46.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.409+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:46.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.410+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:46.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.410+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:21:46.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.410+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:46.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.410+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:46.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.410+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:46.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.410+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:46.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.410+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:46.421 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:46.410+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14713 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:46.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:46.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:46.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:46.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:46.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:46.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:21:46.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:21:46.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:46.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:46.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63e005c7c0 2021-03-09T09:21:46.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:46.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:46.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017c30 con 0x7f63e005c7c0 2021-03-09T09:21:46.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:21:46.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:46.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5474 cs=0 l=1 rev1=1 rx=0x7f63f0072c60 tx=0x7f63f0072aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:46.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:21:46.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.411+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:46.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.414+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0072d40 con 0x7f63e005c7c0 2021-03-09T09:21:46.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.414+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:46.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.414+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:46.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.414+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:46.428 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:46.428 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:46.428 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:46.428 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:46.428 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:46.429 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:46.429 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:46.429 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:46.429 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:46.429 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:46.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.414+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005ee70 con 0x7f63e005c7c0 2021-03-09T09:21:46.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.414+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:46.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.414+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005dc00 con 0x7f63e005c7c0 2021-03-09T09:21:46.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.414+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:21:46.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:46.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:46.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:47.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:47.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:47.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:47.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:48.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:48.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:48.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:48.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:49.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:49.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:49.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:49.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:49.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:49.726+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:50.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:50.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:50.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:50.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:51.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:51.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5474 cs=0 l=1 rev1=1 rx=0x7f63f0072c60 tx=0x7f63f0072aa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:51.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5474 cs=0 l=1 rev1=1 rx=0x7f63f0072c60 tx=0x7f63f0072aa0).stop 2021-03-09T09:21:51.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:51.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:51.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:51.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:51.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:21:51.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:51.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:51.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:51.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:51.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:51.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:51.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:51.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f490 con 0x7f63e00070a0 2021-03-09T09:21:51.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:51.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:21:51.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63e0064730 2021-03-09T09:21:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:51.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:51.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.411+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.412+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.412+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:21:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.412+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.412+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:51.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.412+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.412+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:51.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.412+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:51.425 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14751 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:51.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:21:51.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:21:51.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00304b0 con 0x7f63e0064730 2021-03-09T09:21:51.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:51.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:51.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014780 con 0x7f63e0064730 2021-03-09T09:21:51.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:21:51.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:51.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:21:51.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:51.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.413+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=5485 cs=0 l=1 rev1=1 rx=0x7f63f404cf50 tx=0x7f63f4072d10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:51.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.414+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009350 con 0x7f63e0064730 2021-03-09T09:21:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.414+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:51.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.414+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.414+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:51.437 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:51.438 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:51.438 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:51.439 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:51.439 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:51.439 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:51.440 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:51.440 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:51.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.414+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405bc80 con 0x7f63e0064730 2021-03-09T09:21:51.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.414+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:51.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.414+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:21:51.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.416+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0064730 2021-03-09T09:21:51.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:51.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:51.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:52.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:52.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:52.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:52.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:52.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:52.592+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:53.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:53.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:53.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:53.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:54.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:54.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:54.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:54.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:54.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:54.727+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:21:55.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:55.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:55.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:55.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:56.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:21:56.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:21:56.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=5485 cs=0 l=1 rev1=1 rx=0x7f63f404cf50 tx=0x7f63f4072d10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:21:56.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=5485 cs=0 l=1 rev1=1 rx=0x7f63f404cf50 tx=0x7f63f4072d10).stop 2021-03-09T09:21:56.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:56.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:21:56.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:21:56.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=CLOSED pgs=5485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:21:56.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:21:56.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:56.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:21:56.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:56.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:21:56.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:21:56.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:21:56.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:56.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002c20 con 0x7f63e00070a0 2021-03-09T09:21:56.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:56.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067950 con 0x7f6400145bd0 2021-03-09T09:21:56.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:21:56.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e005c7c0 2021-03-09T09:21:56.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:21:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:21:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.413+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.414+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:21:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.414+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:21:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.414+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:21:56.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.414+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:21:56.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.414+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:21:56.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.414+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:21:56.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:21:56.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:21:56.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:21:56.426 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14785 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:21:56.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:21:56.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:56.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:56.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:21:56.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:21:56.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:21:56.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:21:56.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.415+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:21:56.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:56.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010db0 con 0x7f63e005c7c0 2021-03-09T09:21:56.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:21:56.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:21:56.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002d60 con 0x7f63e005c7c0 2021-03-09T09:21:56.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:21:56.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:56.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:21:56.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 rx=0x7f63e006a710 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:21:56.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.416+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:21:56.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e005c7c0 2021-03-09T09:21:56.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.417+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:21:56.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.417+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:21:56.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.417+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:21:56.433 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:21:56.433 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:21:56.433 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:56.434 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:21:56.434 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:21:56.434 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:21:56.434 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:21:56.435 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:21:56.435 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:21:56.435 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:21:56.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc006480 con 0x7f63e005c7c0 2021-03-09T09:21:56.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.417+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:21:56.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.417+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:21:56.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.418+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63e005c7c0 2021-03-09T09:21:56.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:56.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:56.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:57.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:57.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:57.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:57.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:58.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:58.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:58.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:58.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:59.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:59.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:21:59.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:59.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:21:59.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:21:59.728+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:00.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:00.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:00.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:00.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:01.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:01.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:01.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 rx=0x7f63e006a710 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:01.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5503 cs=0 l=1 rev1=1 rx=0x7f63e006a710 tx=0x7f63fc009990).stop 2021-03-09T09:22:01.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:01.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:01.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:01.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:01.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:22:01.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:01.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:01.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:01.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:01.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:01.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:01.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:01.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63e00070a0 2021-03-09T09:22:01.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:01.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9f0 con 0x7f6400145bd0 2021-03-09T09:22:01.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:01.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63e0064730 2021-03-09T09:22:01.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:01.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:01.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:01.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:01.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.416+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:01.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.417+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:01.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.417+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:22:01.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.417+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:01.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.417+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:01.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.417+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:01.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.417+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:01.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.417+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:01.427 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14827 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:01.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:01.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:01.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:01.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:01.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:01.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:22:01.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:22:01.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:01.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:01.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e440 con 0x7f63e0064730 2021-03-09T09:22:01.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:01.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:01.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00304b0 con 0x7f63e0064730 2021-03-09T09:22:01.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:22:01.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.418+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:01.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.419+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:22:01.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.419+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 secure :-1 s=READY pgs=5520 cs=0 l=1 rev1=1 rx=0x7f63f405b7f0 tx=0x7f63f40093c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:01.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.419+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:01.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.421+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0064730 2021-03-09T09:22:01.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.421+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:01.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.421+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:01.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.421+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:01.433 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:01.433 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:01.434 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:01.434 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:01.434 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:01.434 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:01.434 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:01.435 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:01.435 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:01.435 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:01.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0064730 2021-03-09T09:22:01.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.423+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:01.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.423+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:22:01.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0064730 2021-03-09T09:22:01.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:01.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:01.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:02.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:02.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:02.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:02.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:02.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:02.594+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:03.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:03.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:03.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:03.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:04.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:04.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:04.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:04.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:04.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:04.729+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:05.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:05.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:05.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:05.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:06.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.418+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:06.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.418+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:06.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.418+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 secure :-1 s=READY pgs=5520 cs=0 l=1 rev1=1 rx=0x7f63f405b7f0 tx=0x7f63f40093c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:06.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.418+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 secure :-1 s=READY pgs=5520 cs=0 l=1 rev1=1 rx=0x7f63f405b7f0 tx=0x7f63f40093c0).stop 2021-03-09T09:22:06.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.418+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:06.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.418+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:06.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.418+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:06.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.418+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 unknown :-1 s=CLOSED pgs=5520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:06.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:22:06.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:06.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:06.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:06.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:06.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:06.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:06.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:06.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:22:06.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:06.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a580 con 0x7f6400145bd0 2021-03-09T09:22:06.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:06.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63e005c7c0 2021-03-09T09:22:06.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:06.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:06.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:06.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:06.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.419+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:06.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.420+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:06.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.420+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:22:06.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.420+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:06.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.420+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:06.429 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:06.420+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14865 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:06.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.420+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:06.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.420+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:06.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.420+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:06.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:06.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:06.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:06.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:06.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:06.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:22:06.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:22:06.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:06.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:06.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017c30 con 0x7f63e005c7c0 2021-03-09T09:22:06.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:06.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:06.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63e005c7c0 2021-03-09T09:22:06.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:22:06.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:06.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:22:06.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5532 cs=0 l=1 rev1=1 rx=0x7f63f001b390 tx=0x7f63f0013b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:06.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.421+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:06.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.424+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e005c7c0 2021-03-09T09:22:06.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.424+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:06.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.424+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:06.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.424+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:06.439 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:06.439 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:06.440 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:06.440 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:06.440 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:06.441 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:06.441 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:06.441 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:06.441 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:06.441 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:06.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e005c7c0 2021-03-09T09:22:06.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.425+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:06.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.425+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:22:06.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001b040 con 0x7f63e005c7c0 2021-03-09T09:22:06.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:06.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:06.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:07.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:07.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:07.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:07.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:08.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:08.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:08.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:08.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:09.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:09.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:09.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:09.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:09.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:09.730+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:10.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:10.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:10.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:10.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.420+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:22:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.420+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:11.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.420+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5532 cs=0 l=1 rev1=1 rx=0x7f63f001b390 tx=0x7f63f0013b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:11.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.420+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5532 cs=0 l=1 rev1=1 rx=0x7f63f001b390 tx=0x7f63f0013b10).stop 2021-03-09T09:22:11.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:11.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:11.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:11.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:11.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:22:11.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:11.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:11.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:11.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:11.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:11.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:11.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:11.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b9b0 con 0x7f63e00070a0 2021-03-09T09:22:11.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:11.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400145bd0 2021-03-09T09:22:11.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:11.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a580 con 0x7f63e0064730 2021-03-09T09:22:11.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:11.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:11.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.421+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:11.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:11.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:22:11.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:11.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:11.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:11.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:11.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:11.432 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14903 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:11.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:11.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.422+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:11.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:11.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:11.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:11.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:11.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:22:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:22:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:11.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:11.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00304b0 con 0x7f63e0064730 2021-03-09T09:22:11.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:11.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:11.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017c30 con 0x7f63e0064730 2021-03-09T09:22:11.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:22:11.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:11.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:22:11.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001c8d0 secure :-1 s=READY pgs=5543 cs=0 l=1 rev1=1 rx=0x7f63f0055eb0 tx=0x7f63f004d110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:11.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.423+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:11.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0072d40 con 0x7f63e0064730 2021-03-09T09:22:11.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.425+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:11.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.425+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:11.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.425+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:11.439 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:11.440 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:11.440 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:11.440 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:11.440 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:11.441 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:11.441 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:11.441 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:11.441 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:11.441 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:11.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63e0064730 2021-03-09T09:22:11.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.425+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:11.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.425+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:22:11.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0048400 con 0x7f63e0064730 2021-03-09T09:22:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:11.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:12.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:12.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:12.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:12.596+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:13.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:13.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:13.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:13.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:14.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:14.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:14.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:14.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:14.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:14.731+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:15.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:15.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:15.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:15.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:16.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:16.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:16.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001c8d0 secure :-1 s=READY pgs=5543 cs=0 l=1 rev1=1 rx=0x7f63f0055eb0 tx=0x7f63f004d110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:16.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001c8d0 secure :-1 s=READY pgs=5543 cs=0 l=1 rev1=1 rx=0x7f63f0055eb0 tx=0x7f63f004d110).stop 2021-03-09T09:22:16.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:16.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:16.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001c8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:16.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001c8d0 unknown :-1 s=CLOSED pgs=5543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:16.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:22:16.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:16.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:16.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:16.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:16.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:16.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:16.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:16.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010310 con 0x7f63e00070a0 2021-03-09T09:22:16.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:16.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016a70 con 0x7f6400145bd0 2021-03-09T09:22:16.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:16.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009100 con 0x7f63e005c7c0 2021-03-09T09:22:16.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:16.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:16.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:16.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:16.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.423+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:16.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.424+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:16.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.424+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:22:16.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.424+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:16.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.424+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:16.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.424+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:16.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.424+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:16.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.424+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:16.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:16.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:16.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:16.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:16.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:16.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:22:16.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:22:16.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:16.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:16.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:22:16.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:16.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:16.438 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14942 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:16.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00304b0 con 0x7f63e005c7c0 2021-03-09T09:22:16.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:22:16.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.425+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:16.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.426+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:22:16.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.426+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:16.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f001cf20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:16.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.427+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63e005c7c0 2021-03-09T09:22:16.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.427+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:16.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.427+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:16.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.427+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:16.441 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:16.442 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:16.442 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:16.442 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:16.442 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:16.442 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:16.443 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:16.443 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:16.443 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:16.443 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:16.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.429+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63e005c7c0 2021-03-09T09:22:16.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.429+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:16.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.429+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:22:16.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0072cd0 con 0x7f63e005c7c0 2021-03-09T09:22:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:16.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:17.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:17.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:17.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:17.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:18.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:18.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:18.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:18.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:19.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:19.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:19.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:19.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:19.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:19.732+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:20.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:20.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:20.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:20.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:21.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:21.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:21.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f001cf20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:21.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5560 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f001cf20).stop 2021-03-09T09:22:21.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:21.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:21.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:21.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:21.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:22:21.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:21.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:21.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:21.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:21.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:21.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:21.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:21.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e00070a0 2021-03-09T09:22:21.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:21.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078110 con 0x7f6400145bd0 2021-03-09T09:22:21.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:21.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63e0064730 2021-03-09T09:22:21.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:21.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:21.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:21.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:21.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:21.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:21.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:22:21.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:21.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.426+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:21.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:21.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:21.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:21.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:21.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:21.438 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 14980 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:21.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:21.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:21.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:21.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:22:21.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:22:21.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:21.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:21.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00304b0 con 0x7f63e0064730 2021-03-09T09:22:21.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:21.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:21.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e440 con 0x7f63e0064730 2021-03-09T09:22:21.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:22:21.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.427+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:21.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.428+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 secure :-1 s=READY pgs=5570 cs=0 l=1 rev1=1 rx=0x7f63f4048a20 tx=0x7f63f405b4e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:21.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.428+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:22:21.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.428+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:21.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048a90 con 0x7f63e0064730 2021-03-09T09:22:21.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.430+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:21.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.430+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:21.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.430+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:21.444 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:21.444 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:21.444 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:21.445 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:21.445 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:21.445 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:21.445 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:21.445 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:21.446 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:21.446 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:21.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e0064730 2021-03-09T09:22:21.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.431+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:21.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.431+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:22:21.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.432+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048a90 con 0x7f63e0064730 2021-03-09T09:22:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:21.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:22.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:22.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:22.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:22.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:22.597+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:23.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:23.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:23.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:23.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:24.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:24.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:24.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:24.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:24.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:24.733+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:25.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:25.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:25.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:25.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:26.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:26.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:26.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 secure :-1 s=READY pgs=5570 cs=0 l=1 rev1=1 rx=0x7f63f4048a20 tx=0x7f63f405b4e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:26.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 secure :-1 s=READY pgs=5570 cs=0 l=1 rev1=1 rx=0x7f63f4048a20 tx=0x7f63f405b4e0).stop 2021-03-09T09:22:26.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:26.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:26.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000d9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:26.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000d9f0 unknown :-1 s=CLOSED pgs=5570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:26.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:22:26.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:26.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:26.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:26.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:26.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:26.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:26.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:26.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:22:26.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:26.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b9b0 con 0x7f6400145bd0 2021-03-09T09:22:26.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:26.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067db0 con 0x7f63e005c7c0 2021-03-09T09:22:26.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:26.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:26.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.428+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:26.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:26.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:22:26.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:26.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:26.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:26.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:26.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:26.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:26.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:26.439 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:26.429+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15019 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:26.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:26.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:26.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:26.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:26.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:26.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:22:26.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:22:26.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:26.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:26.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012730 con 0x7f63e005c7c0 2021-03-09T09:22:26.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:26.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:26.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:22:26.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:22:26.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.430+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:26.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.431+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5586 cs=0 l=1 rev1=1 rx=0x7f63f0030fd0 tx=0x7f63f0006c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:26.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.431+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:22:26.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.431+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:26.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.432+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0071030 con 0x7f63e005c7c0 2021-03-09T09:22:26.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.432+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:26.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.432+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:26.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.432+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:26.445 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:26.446 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:26.446 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:26.446 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:26.446 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:26.446 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:26.447 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:26.447 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:26.447 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:26.447 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:26.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.432+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0071030 con 0x7f63e005c7c0 2021-03-09T09:22:26.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.432+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:26.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.432+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:22:26.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e005c7c0 2021-03-09T09:22:26.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:26.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:26.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:27.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:27.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:27.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:27.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:28.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:28.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:28.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:28.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:29.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:29.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:29.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:29.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:29.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:29.734+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:30.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:30.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:30.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:30.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:22:31.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:31.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5586 cs=0 l=1 rev1=1 rx=0x7f63f0030fd0 tx=0x7f63f0006c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:31.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5586 cs=0 l=1 rev1=1 rx=0x7f63f0030fd0 tx=0x7f63f0006c30).stop 2021-03-09T09:22:31.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:31.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:31.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:31.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:31.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:22:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016fd0 con 0x7f63e00070a0 2021-03-09T09:22:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00140b0 con 0x7f6400145bd0 2021-03-09T09:22:31.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:31.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63e0064730 2021-03-09T09:22:31.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:31.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:31.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:31.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:31.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.430+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:31.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.431+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:31.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.431+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:22:31.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.431+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:31.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.431+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:31.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.431+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:31.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.431+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:31.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.431+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:31.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:31.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:31.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:31.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:31.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:31.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:22:31.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:22:31.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:31.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:31.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0064730 2021-03-09T09:22:31.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:31.444 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15057 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:31.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:31.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012730 con 0x7f63e0064730 2021-03-09T09:22:31.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:22:31.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.432+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:31.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.433+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5599 cs=0 l=1 rev1=1 rx=0x7f63f003fdf0 tx=0x7f63f0071110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:31.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.433+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:22:31.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.433+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:31.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.435+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63e0064730 2021-03-09T09:22:31.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.435+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:31.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.435+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:31.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.435+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:31.448 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:31.448 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:31.448 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:31.449 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:31.449 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:31.449 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:31.449 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:31.449 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:31.450 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:31.450 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:31.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.437+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0019140 con 0x7f63e0064730 2021-03-09T09:22:31.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.437+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:31.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.438+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:22:31.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027760 con 0x7f63e0064730 2021-03-09T09:22:31.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:31.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:31.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:32.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:32.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:32.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:32.598+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:33.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:33.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:33.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:33.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:34.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:34.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:34.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:34.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:34.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:34.735+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:35.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:35.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:35.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:35.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:36.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:36.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:36.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5599 cs=0 l=1 rev1=1 rx=0x7f63f003fdf0 tx=0x7f63f0071110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:36.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5599 cs=0 l=1 rev1=1 rx=0x7f63f003fdf0 tx=0x7f63f0071110).stop 2021-03-09T09:22:36.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:36.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:36.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:36.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=CLOSED pgs=5599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:22:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:36.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:36.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:36.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:36.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63e00070a0 2021-03-09T09:22:36.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:36.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078110 con 0x7f6400145bd0 2021-03-09T09:22:36.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:36.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a580 con 0x7f63e005c7c0 2021-03-09T09:22:36.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:36.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:36.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:36.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:36.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:36.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.433+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:36.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.434+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:22:36.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.434+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:36.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.434+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:36.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.434+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:36.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.434+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:36.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.434+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:36.445 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:36.434+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15095 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:36.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:22:36.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:22:36.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:36.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:36.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d810 con 0x7f63e005c7c0 2021-03-09T09:22:36.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:36.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:36.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:22:36.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:22:36.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:36.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:22:36.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:36.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5616 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f001cf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:36.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003fbc0 con 0x7f63e005c7c0 2021-03-09T09:22:36.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.440+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:36.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.440+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:36.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.440+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:36.451 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:36.451 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:36.452 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:36.452 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:36.452 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:36.452 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:36.452 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:36.453 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:36.453 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:36.453 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:36.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.441+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00405d0 con 0x7f63e005c7c0 2021-03-09T09:22:36.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.441+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:36.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.441+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:22:36.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63e005c7c0 2021-03-09T09:22:36.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:36.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:36.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:37.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:37.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:37.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:37.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:38.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:38.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:38.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:38.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:39.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:39.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:39.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:39.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:39.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:39.736+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:40.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:40.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:40.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:40.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:41.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5616 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f001cf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5616 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f001cf90).stop 2021-03-09T09:22:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:22:41.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:41.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:41.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:41.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b6c0 con 0x7f63e00070a0 2021-03-09T09:22:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:41.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9f0 con 0x7f6400145bd0 2021-03-09T09:22:41.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:41.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00140b0 con 0x7f63e0064730 2021-03-09T09:22:41.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:41.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:41.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:41.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.435+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:41.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.436+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:41.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:41.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.436+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:22:41.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.436+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:41.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.436+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:41.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.436+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:41.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.436+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:41.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.436+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:41.447 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15130 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:41.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:41.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:41.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:41.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:41.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:41.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:22:41.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:22:41.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:41.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:41.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63e0064730 2021-03-09T09:22:41.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:41.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:41.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d810 con 0x7f63e0064730 2021-03-09T09:22:41.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:22:41.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.437+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.438+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5629 cs=0 l=1 rev1=1 rx=0x7f63f004e510 tx=0x7f63f0031f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.439+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:22:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.439+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.441+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005ec60 con 0x7f63e0064730 2021-03-09T09:22:41.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.441+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:41.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.441+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:41.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.441+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:41.453 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:41.453 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:41.454 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:41.454 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:41.454 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:41.454 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:41.454 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:41.455 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:41.455 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:41.455 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:41.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.441+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005fdb0 con 0x7f63e0064730 2021-03-09T09:22:41.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.441+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:41.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.441+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:22:41.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00312f0 con 0x7f63e0064730 2021-03-09T09:22:41.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:41.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:41.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:42.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:42.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:42.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:42.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:42.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:42.599+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:43.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:43.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:43.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:43.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:44.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:44.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:44.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:44.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:44.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:44.737+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:45.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:45.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:45.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:46.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:22:46.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:46.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5629 cs=0 l=1 rev1=1 rx=0x7f63f004e510 tx=0x7f63f0031f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:46.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5629 cs=0 l=1 rev1=1 rx=0x7f63f004e510 tx=0x7f63f0031f50).stop 2021-03-09T09:22:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:46.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=CLOSED pgs=5629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:46.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:22:46.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:46.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:46.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:46.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:46.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:46.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:46.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:46.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a580 con 0x7f63e00070a0 2021-03-09T09:22:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400145bd0 2021-03-09T09:22:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078110 con 0x7f63e005c7c0 2021-03-09T09:22:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:22:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:46.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.438+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:46.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:46.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:46.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:46.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:46.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:22:46.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:22:46.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:46.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:46.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016410 con 0x7f63e005c7c0 2021-03-09T09:22:46.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:46.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:46.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010db0 con 0x7f63e005c7c0 2021-03-09T09:22:46.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:22:46.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:46.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5637 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:46.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:22:46.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:46.453 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:46.439+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15171 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:46.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065310 con 0x7f63e005c7c0 2021-03-09T09:22:46.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.440+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.440+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.440+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:46.457 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:46.457 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:46.457 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:46.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065310 con 0x7f63e005c7c0 2021-03-09T09:22:46.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.440+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:46.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.440+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:22:46.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.441+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:22:46.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:46.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:46.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:47.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:47.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:47.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:47.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:48.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:48.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:48.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:48.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:49.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:49.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:49.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:49.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:49.738+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:50.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:50.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:50.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:50.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:51.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5637 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5637 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc04a740).stop 2021-03-09T09:22:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:51.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:51.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:22:51.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:51.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:51.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:51.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:51.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:51.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:51.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:51.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:22:51.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:51.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b9b0 con 0x7f6400145bd0 2021-03-09T09:22:51.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:51.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b6c0 con 0x7f63e0064730 2021-03-09T09:22:51.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:51.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:51.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:51.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:51.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:51.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:22:51.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:51.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:51.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.440+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:51.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:51.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:51.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:51.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:51.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:51.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:51.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:51.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:51.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:22:51.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:22:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017c30 con 0x7f63e0064730 2021-03-09T09:22:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00304b0 con 0x7f63e0064730 2021-03-09T09:22:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:22:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.441+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:51.456 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:51.442+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15211 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.442+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:22:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.442+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.442+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5652 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f405b450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.444+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402aec0 con 0x7f63e0064730 2021-03-09T09:22:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.444+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.444+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.444+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:51.460 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.444+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f402aec0 con 0x7f63e0064730 2021-03-09T09:22:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.444+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:51.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.444+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:22:51.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.445+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e0064730 2021-03-09T09:22:51.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:51.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:51.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:52.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:52.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:52.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:52.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:52.601+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:53.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:53.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:53.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:53.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:54.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:54.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:54.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:54.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:54.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:54.739+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:22:55.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:55.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:55.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:55.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:56.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:22:56.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:22:56.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5652 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f405b450).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:22:56.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5652 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f405b450).stop 2021-03-09T09:22:56.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:56.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:22:56.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:22:56.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=CLOSED pgs=5652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:22:56.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:22:56.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:56.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:22:56.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:56.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:22:56.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:22:56.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:22:56.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:56.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f63e00070a0 2021-03-09T09:22:56.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:56.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069660 con 0x7f6400145bd0 2021-03-09T09:22:56.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:22:56.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00140b0 con 0x7f63e005c7c0 2021-03-09T09:22:56.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:22:56.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:22:56.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:56.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:22:56.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:22:56.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:22:56.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:22:56.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:22:56.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.443+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:22:56.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.444+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:22:56.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.444+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:22:56.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.444+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:22:56.454 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:22:56.444+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15248 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:22:56.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:22:56.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:22:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:22:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:22:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:22:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:22:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:22:56.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:22:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:22:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016410 con 0x7f63e005c7c0 2021-03-09T09:22:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:22:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.445+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.446+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5670 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc03f2f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:22:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.446+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:22:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.446+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:22:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.446+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03cb40 con 0x7f63e005c7c0 2021-03-09T09:22:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.446+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:22:56.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.446+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:22:56.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.446+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:22:56.462 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:22:56.462 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:22:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:22:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:22:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:22:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:22:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:22:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:22:56.464 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:22:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.447+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03cb40 con 0x7f63e005c7c0 2021-03-09T09:22:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.447+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:22:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.447+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:22:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63e005c7c0 2021-03-09T09:22:56.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:56.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:56.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:57.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:57.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:57.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:57.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:58.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:58.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:58.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:58.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:59.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:59.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:22:59.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:59.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:22:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:22:59.740+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:00.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:00.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:00.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:00.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:01.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:01.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:01.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5670 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc03f2f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:01.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5670 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc03f2f0).stop 2021-03-09T09:23:01.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:01.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:01.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:23:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:01.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:01.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:01.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:01.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:01.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068fe0 con 0x7f63e00070a0 2021-03-09T09:23:01.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:01.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078110 con 0x7f6400145bd0 2021-03-09T09:23:01.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:01.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a580 con 0x7f63e0064730 2021-03-09T09:23:01.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:01.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:01.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:01.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:01.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:01.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:01.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:23:01.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:01.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.446+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:01.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:01.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:01.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:01.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:01.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:01.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:01.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:01.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:01.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:23:01.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:23:01.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:01.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:01.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013250 con 0x7f63e0064730 2021-03-09T09:23:01.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:01.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:01.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017c30 con 0x7f63e0064730 2021-03-09T09:23:01.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:23:01.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.447+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:01.461 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:01.448+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15288 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:01.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.448+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:23:01.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.448+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5678 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4061850).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:01.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.448+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:01.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072e50 con 0x7f63e0064730 2021-03-09T09:23:01.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.450+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:01.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.450+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:01.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.450+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:01.464 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:01.464 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:01.464 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:01.464 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:01.464 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:01.465 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:01.465 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:01.465 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:01.465 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:01.465 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:01.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072e50 con 0x7f63e0064730 2021-03-09T09:23:01.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.451+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:01.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.451+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:23:01.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40611a0 con 0x7f63e0064730 2021-03-09T09:23:01.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:01.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:01.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:02.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:02.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:02.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:02.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:02.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:02.603+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:03.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:03.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:03.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:03.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:04.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:04.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:04.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:04.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:04.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:04.741+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:05.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:05.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:05.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:05.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:06.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:06.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:06.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5678 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4061850).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:06.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 secure :-1 s=READY pgs=5678 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4061850).stop 2021-03-09T09:23:06.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:06.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:06.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0065790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:06.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0065790 unknown :-1 s=CLOSED pgs=5678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:06.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:23:06.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:06.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:06.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:06.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:06.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:06.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63e00070a0 2021-03-09T09:23:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b9b0 con 0x7f6400145bd0 2021-03-09T09:23:06.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:06.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b4c0 con 0x7f63e005c7c0 2021-03-09T09:23:06.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:06.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:06.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.448+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:06.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.449+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:06.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.449+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:06.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.449+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:06.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.449+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:23:06.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.449+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:06.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.449+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:06.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.449+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:06.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.449+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:06.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.449+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:06.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:06.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:06.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:06.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:06.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:06.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:23:06.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:23:06.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:06.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:06.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a700 con 0x7f63e005c7c0 2021-03-09T09:23:06.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:06.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:06.461 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15325 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:06.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63e005c7c0 2021-03-09T09:23:06.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:23:06.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:06.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:23:06.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5693 cs=0 l=1 rev1=1 rx=0x7f63e0069830 tx=0x7f63f004d190).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:06.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.450+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:06.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.453+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63e005c7c0 2021-03-09T09:23:06.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.453+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:06.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.453+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:06.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.453+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:06.465 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:06.465 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:06.465 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:06.465 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:06.466 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:06.466 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:06.466 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:06.466 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:06.466 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:06.467 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:06.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.454+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e005c7c0 2021-03-09T09:23:06.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.454+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:06.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.454+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:23:06.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e005c7c0 2021-03-09T09:23:06.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:06.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:06.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:07.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:07.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:07.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:07.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:08.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:08.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:08.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:08.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:09.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:09.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:09.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:09.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:09.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:09.742+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:10.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:10.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:10.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:10.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:11.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:23:11.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:11.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5693 cs=0 l=1 rev1=1 rx=0x7f63e0069830 tx=0x7f63f004d190).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:11.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5693 cs=0 l=1 rev1=1 rx=0x7f63e0069830 tx=0x7f63f004d190).stop 2021-03-09T09:23:11.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:11.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:11.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:11.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:11.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:23:11.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:11.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009100 con 0x7f63e00070a0 2021-03-09T09:23:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001950 con 0x7f6400145bd0 2021-03-09T09:23:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078110 con 0x7f63e0064730 2021-03-09T09:23:11.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.450+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:11.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.451+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:11.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.451+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:11.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.451+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:11.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.451+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.451+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.451+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:23:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:11.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:11.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:11.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:11.561 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15365 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:23:11.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:23:11.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:11.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:11.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e440 con 0x7f63e0064730 2021-03-09T09:23:11.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:11.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:11.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a700 con 0x7f63e0064730 2021-03-09T09:23:11.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.453+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:23:11.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.453+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:11.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.453+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022a60 secure :-1 s=READY pgs=5707 cs=0 l=1 rev1=1 rx=0x7f63f004cdf0 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:11.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.453+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:23:11.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.453+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:11.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0064730 2021-03-09T09:23:11.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.455+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:11.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.455+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:11.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.455+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:11.568 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:11.569 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:11.570 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:11.570 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:11.570 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:11.570 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:11.570 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:11.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.456+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0064730 2021-03-09T09:23:11.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.456+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:11.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.456+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:23:11.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0030430 con 0x7f63e0064730 2021-03-09T09:23:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:11.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:11.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:12.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:12.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:12.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:12.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:12.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:12.604+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:13.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:13.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:13.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:13.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:14.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:14.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:14.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:14.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:14.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:14.743+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:15.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:15.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:15.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:15.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0022a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:16.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0022a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:16.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022a60 secure :-1 s=READY pgs=5707 cs=0 l=1 rev1=1 rx=0x7f63f004cdf0 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:16.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022a60 secure :-1 s=READY pgs=5707 cs=0 l=1 rev1=1 rx=0x7f63f004cdf0 tx=0x7f63f0000c30).stop 2021-03-09T09:23:16.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:16.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:16.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0022a60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:16.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0022a60 unknown :-1 s=CLOSED pgs=5707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:16.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:23:16.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:16.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:16.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:16.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:16.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:16.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:16.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:16.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f240 con 0x7f63e00070a0 2021-03-09T09:23:16.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:16.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024cc0 con 0x7f6400145bd0 2021-03-09T09:23:16.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:16.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:23:16.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:16.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:16.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:16.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:16.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:16.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:16.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:23:16.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:16.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.453+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:16.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:16.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:16.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:16.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:16.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:16.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:16.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:16.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:16.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:23:16.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:23:16.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:16.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:16.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00304b0 con 0x7f63e005c7c0 2021-03-09T09:23:16.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:16.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:16.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:23:16.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:23:16.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.454+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:16.467 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:16.455+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15392 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:16.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.455+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:23:16.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.455+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5719 cs=0 l=1 rev1=1 rx=0x7f63fc009860 tx=0x7f63fc046b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:16.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.455+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:16.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.456+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0711a0 con 0x7f63e005c7c0 2021-03-09T09:23:16.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.456+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:16.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.456+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:16.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.456+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:16.470 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:16.470 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:16.470 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:16.470 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:16.470 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:16.471 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:16.471 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:16.471 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:16.471 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:16.471 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:16.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.456+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:23:16.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.456+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:16.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.456+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:23:16.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:23:16.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:16.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:16.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:17.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:17.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:17.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:17.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:18.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:18.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:18.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:18.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:19.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:19.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:19.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:19.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:19.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:19.744+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:20.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:20.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:20.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:20.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:21.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:21.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:21.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5719 cs=0 l=1 rev1=1 rx=0x7f63fc009860 tx=0x7f63fc046b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:21.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5719 cs=0 l=1 rev1=1 rx=0x7f63fc009860 tx=0x7f63fc046b50).stop 2021-03-09T09:23:21.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:21.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:21.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:21.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:21.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:23:21.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:21.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:21.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:21.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:21.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:21.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:21.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:21.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d680 con 0x7f63e00070a0 2021-03-09T09:23:21.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:21.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:23:21.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:21.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024cc0 con 0x7f63e0064730 2021-03-09T09:23:21.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:21.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:21.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.455+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.456+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.456+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.456+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.456+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:23:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.456+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.456+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:21.465 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15442 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:21.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:21.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:23:21.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:23:21.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:21.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:21.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63e0064730 2021-03-09T09:23:21.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:21.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:21.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013250 con 0x7f63e0064730 2021-03-09T09:23:21.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:23:21.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.457+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:21.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.458+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5735 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f40093e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.459+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:23:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.459+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63e0064730 2021-03-09T09:23:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.460+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.460+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.460+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:21.474 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.462+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63e0064730 2021-03-09T09:23:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.462+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:21.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.462+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:23:21.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.463+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4072d80 con 0x7f63e0064730 2021-03-09T09:23:21.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:21.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:21.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:22.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:22.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:22.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:22.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:22.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:22.605+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:23.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:23.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:23.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:23.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:24.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:24.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:24.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:24.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:24.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:24.744+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:25.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:25.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:25.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:25.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:26.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:26.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:26.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5735 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f40093e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:26.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5735 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f40093e0).stop 2021-03-09T09:23:26.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:26.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:26.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:26.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=CLOSED pgs=5735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:26.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:23:26.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:26.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:26.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:26.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:26.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:26.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:26.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:26.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024540 con 0x7f63e00070a0 2021-03-09T09:23:26.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:26.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078270 con 0x7f6400145bd0 2021-03-09T09:23:26.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:26.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c490 con 0x7f63e005c7c0 2021-03-09T09:23:26.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:26.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:26.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:26.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:26.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:26.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.457+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:26.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:23:26.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:26.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:26.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:26.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:26.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:26.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:26.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:26.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:26.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:26.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:26.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:23:26.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:23:26.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:26.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:26.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e2e0 con 0x7f63e005c7c0 2021-03-09T09:23:26.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:26.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:26.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63e005c7c0 2021-03-09T09:23:26.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:23:26.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.458+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:26.472 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:26.459+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15479 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:26.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.459+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:23:26.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.459+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5747 cs=0 l=1 rev1=1 rx=0x7f63f400b160 tx=0x7f63f404b9b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:26.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.459+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:26.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e005c7c0 2021-03-09T09:23:26.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.461+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:26.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.461+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:26.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.461+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:26.476 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:26.476 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:26.476 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:26.476 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:26.476 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:26.477 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:26.477 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:26.478 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:26.478 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:26.478 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:26.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.463+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63e005c7c0 2021-03-09T09:23:26.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.463+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:26.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.463+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:23:26.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e005c7c0 2021-03-09T09:23:26.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:26.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:26.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:27.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:27.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:27.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:27.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:28.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:28.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:28.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:28.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:29.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:29.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:29.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:29.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:29.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:29.745+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:30.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:30.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:31.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:31.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:31.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5747 cs=0 l=1 rev1=1 rx=0x7f63f400b160 tx=0x7f63f404b9b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:31.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 secure :-1 s=READY pgs=5747 cs=0 l=1 rev1=1 rx=0x7f63f400b160 tx=0x7f63f404b9b0).stop 2021-03-09T09:23:31.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:31.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:31.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:31.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c5c0 unknown :-1 s=CLOSED pgs=5747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:31.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:23:31.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:31.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:31.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:31.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:31.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:31.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:31.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:31.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a600 con 0x7f63e00070a0 2021-03-09T09:23:31.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:31.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078110 con 0x7f6400145bd0 2021-03-09T09:23:31.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:31.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:23:31.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:31.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:31.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:31.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:31.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.459+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:31.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.460+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:31.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.460+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:23:31.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.460+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:31.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.460+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.460+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.460+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.460+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:31.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:31.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:31.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:31.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:23:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:23:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:31.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013880 con 0x7f63e0064730 2021-03-09T09:23:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00304b0 con 0x7f63e0064730 2021-03-09T09:23:31.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:23:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:31.474 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:31.461+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15519 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:31.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.462+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5761 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc066fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:31.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.462+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:23:31.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.462+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:31.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e0064730 2021-03-09T09:23:31.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.467+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:31.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.467+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.467+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:31.478 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:31.479 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d2f0 con 0x7f63e0064730 2021-03-09T09:23:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.467+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.467+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:23:31.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f63e0064730 2021-03-09T09:23:31.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:31.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:31.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:32.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:32.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:32.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:32.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:32.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:32.606+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:33.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:33.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:33.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:33.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:34.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:34.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:34.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:34.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:34.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:34.746+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:35.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:35.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:35.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:35.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:23:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:36.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5761 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc066fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:36.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5761 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc066fa0).stop 2021-03-09T09:23:36.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:36.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:36.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:36.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=CLOSED pgs=5761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:23:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:36.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:36.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:36.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:36.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:36.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009100 con 0x7f63e00070a0 2021-03-09T09:23:36.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:36.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400145bd0 2021-03-09T09:23:36.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:36.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009300 con 0x7f63e005c7c0 2021-03-09T09:23:36.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.461+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:23:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:36.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:36.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:36.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.462+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:36.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:36.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:23:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:23:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:36.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:23:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013880 con 0x7f63e005c7c0 2021-03-09T09:23:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:23:36.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:36.476 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:36.463+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15556 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.464+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:23:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.464+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5775 cs=0 l=1 rev1=1 rx=0x7f63e005eae0 tx=0x7f63fc075f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.464+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:36.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.465+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63e005c7c0 2021-03-09T09:23:36.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.465+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:36.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.465+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:36.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.465+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:36.479 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:36.479 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:36.479 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:36.479 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:36.479 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:36.480 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:36.480 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:36.480 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:36.480 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:36.480 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:36.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.466+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023590 con 0x7f63e005c7c0 2021-03-09T09:23:36.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.466+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:36.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.466+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:23:36.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63e005c7c0 2021-03-09T09:23:36.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:36.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:36.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:37.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:37.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:37.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:38.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:38.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:38.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:38.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:39.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:39.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:39.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:39.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:39.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:39.747+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:40.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:40.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:40.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:41.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:41.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5775 cs=0 l=1 rev1=1 rx=0x7f63e005eae0 tx=0x7f63fc075f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:41.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5775 cs=0 l=1 rev1=1 rx=0x7f63e005eae0 tx=0x7f63fc075f70).stop 2021-03-09T09:23:41.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:41.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:41.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:41.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=5775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:41.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:23:41.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:41.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:41.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:41.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:41.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:41.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024540 con 0x7f63e00070a0 2021-03-09T09:23:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024cc0 con 0x7f6400145bd0 2021-03-09T09:23:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:41.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d680 con 0x7f63e0064730 2021-03-09T09:23:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:41.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:23:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:41.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.464+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.465+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:41.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.465+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:41.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.465+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:41.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:41.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:41.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:41.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:41.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:41.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:23:41.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:23:41.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:41.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:41.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006a5b0 con 0x7f63e0064730 2021-03-09T09:23:41.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:41.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:41.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e440 con 0x7f63e0064730 2021-03-09T09:23:41.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:23:41.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:41.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:23:41.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5789 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000f7d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:41.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:41.482 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:41.466+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15596 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:41.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000eef0 con 0x7f63e0064730 2021-03-09T09:23:41.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.467+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:41.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.467+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:41.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.467+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:41.484 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:41.484 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:41.484 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:41.484 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:41.484 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:41.485 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:41.485 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:41.485 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:41.485 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:41.486 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:41.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0064730 2021-03-09T09:23:41.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.467+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:41.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.467+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:23:41.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T09:23:41.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:41.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:41.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:42.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:42.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:42.607+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:43.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:43.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:43.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:43.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:44.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:44.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:44.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:44.748+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:45.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:45.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:45.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:45.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:46.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.466+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:23:46.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.466+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:46.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.466+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5789 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000f7d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:46.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.466+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5789 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000f7d0).stop 2021-03-09T09:23:46.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.466+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:46.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.466+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:46.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.466+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:46.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.466+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=CLOSED pgs=5789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:46.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:23:46.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:46.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:46.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:46.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:46.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:46.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:46.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:46.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078110 con 0x7f63e00070a0 2021-03-09T09:23:46.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:46.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f6400145bd0 2021-03-09T09:23:46.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:46.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00692f0 con 0x7f63e005c7c0 2021-03-09T09:23:46.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:46.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:46.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:46.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:46.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:46.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:46.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:23:46.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:46.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:46.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:46.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:46.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.467+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:46.478 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:46.468+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15633 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:46.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:46.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:46.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:46.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:46.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:46.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:23:46.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:23:46.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:46.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:46.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002b9d0 con 0x7f63e005c7c0 2021-03-09T09:23:46.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:46.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:46.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e2e0 con 0x7f63e005c7c0 2021-03-09T09:23:46.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:23:46.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.470+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:46.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:23:46.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068250 secure :-1 s=READY pgs=5810 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4017f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:46.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:46.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009210 con 0x7f63e005c7c0 2021-03-09T09:23:46.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:46.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:46.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:46.485 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:46.485 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:46.485 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:46.485 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:46.485 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:46.486 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:46.486 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:46.486 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:46.486 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:46.486 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:46.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072d80 con 0x7f63e005c7c0 2021-03-09T09:23:46.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:46.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.471+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:23:46.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.472+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4062700 con 0x7f63e005c7c0 2021-03-09T09:23:46.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:46.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:46.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:47.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:47.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:47.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:47.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:48.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:48.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:48.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:48.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:49.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:49.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:49.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:49.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:49.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:49.749+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:50.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:50.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:50.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:50.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:51.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0068250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:23:51.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0068250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:51.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068250 secure :-1 s=READY pgs=5810 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4017f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:51.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068250 secure :-1 s=READY pgs=5810 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4017f40).stop 2021-03-09T09:23:51.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:51.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:51.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0068250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:51.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0068250 unknown :-1 s=CLOSED pgs=5810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:51.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:23:51.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:51.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:51.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:51.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:51.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:51.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:51.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:51.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d680 con 0x7f63e00070a0 2021-03-09T09:23:51.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:51.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024cc0 con 0x7f6400145bd0 2021-03-09T09:23:51.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:51.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024250 con 0x7f63e0064730 2021-03-09T09:23:51.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:51.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.469+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:51.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.470+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:51.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.470+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:51.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.470+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:51.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.470+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:51.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.470+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:23:51.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.470+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:51.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.470+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:51.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:51.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:51.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:51.481 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15671 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:51.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:51.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:51.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:51.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:51.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:51.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:23:51.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:23:51.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:51.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:51.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024540 con 0x7f63e0064730 2021-03-09T09:23:51.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:51.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:51.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T09:23:51.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:23:51.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.471+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:51.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.472+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5818 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc04bd50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:51.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.472+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:23:51.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.472+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:51.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63e0064730 2021-03-09T09:23:51.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.473+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:51.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.473+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:51.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.473+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:51.488 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:51.488 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:51.489 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:51.489 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:51.489 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:51.489 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:51.489 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:51.490 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:51.490 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:51.490 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:51.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e0064730 2021-03-09T09:23:51.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.473+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:51.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.473+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:23:51.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63e0064730 2021-03-09T09:23:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:51.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:52.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:52.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:52.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:52.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:52.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:52.608+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:53.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:53.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:53.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:53.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:54.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:54.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:54.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:54.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:54.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:54.750+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:23:55.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:55.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:55.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:55.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:56.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:23:56.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:23:56.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5818 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc04bd50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:23:56.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 secure :-1 s=READY pgs=5818 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc04bd50).stop 2021-03-09T09:23:56.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:56.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:23:56.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001b570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:23:56.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001b570 unknown :-1 s=CLOSED pgs=5818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:23:56.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:23:56.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:56.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:23:56.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:56.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:23:56.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:23:56.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:23:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015400 con 0x7f63e00070a0 2021-03-09T09:23:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:23:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:23:56.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078110 con 0x7f63e005c7c0 2021-03-09T09:23:56.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.471+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:23:56.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.472+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:23:56.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.472+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:56.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.472+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:23:56.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.472+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:23:56.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.472+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:23:56.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.472+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:23:56.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.472+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:23:56.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:23:56.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:23:56.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:23:56.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:23:56.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:23:56.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:56.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:56.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:23:56.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:23:56.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:23:56.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:23:56.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:23:56.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:56.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017c30 con 0x7f63e005c7c0 2021-03-09T09:23:56.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:23:56.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:23:56.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002b9d0 con 0x7f63e005c7c0 2021-03-09T09:23:56.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:23:56.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:56.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5830 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4034220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:23:56.487 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:23:56.473+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15710 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:23:56.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.474+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:23:56.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.474+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:23:56.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.475+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63e005c7c0 2021-03-09T09:23:56.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.475+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:23:56.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.475+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:23:56.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.475+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:23:56.489 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:23:56.490 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:23:56.490 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:56.490 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:23:56.490 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:23:56.490 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:23:56.491 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:23:56.491 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:23:56.491 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:23:56.491 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:23:56.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.475+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404ba20 con 0x7f63e005c7c0 2021-03-09T09:23:56.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.475+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:23:56.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.475+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:23:56.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.475+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4072d80 con 0x7f63e005c7c0 2021-03-09T09:23:56.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:56.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:56.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:57.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:57.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:58.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:57.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:58.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:58.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:58.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:58.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:59.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:59.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:23:59.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:59.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:23:59.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:23:59.751+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:00.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:00.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:00.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:01.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:01.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:01.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5830 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4034220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5830 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4034220).stop 2021-03-09T09:24:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=5830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:24:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:01.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:01.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:01.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d0f0 con 0x7f63e00070a0 2021-03-09T09:24:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027c30 con 0x7f6400145bd0 2021-03-09T09:24:01.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:01.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f0c0 con 0x7f63e0064730 2021-03-09T09:24:01.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:01.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:01.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:01.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:01.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.474+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.475+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.475+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:24:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.475+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.475+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.475+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.475+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.475+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:24:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:24:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T09:24:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:01.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006a5b0 con 0x7f63e0064730 2021-03-09T09:24:01.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:24:01.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:01.489 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:01.476+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15750 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:01.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.477+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:24:01.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.477+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:01.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.477+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018390 secure :-1 s=READY pgs=5844 cs=0 l=1 rev1=1 rx=0x7f63e0065dd0 tx=0x7f63f0006690).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:01.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.478+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0064730 2021-03-09T09:24:01.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.478+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:01.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.478+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:01.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.478+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:01.492 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:01.492 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:01.492 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:01.494 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:01.495 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:01.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0064730 2021-03-09T09:24:01.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.480+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:01.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.480+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:24:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T09:24:01.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:01.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:01.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:02.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:02.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:02.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:02.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:02.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:02.610+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:03.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:03.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:03.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:03.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:04.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:04.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:04.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:04.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:04.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:04.752+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:05.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:05.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:05.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:05.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:06.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0018390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:06.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0018390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:06.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018390 secure :-1 s=READY pgs=5844 cs=0 l=1 rev1=1 rx=0x7f63e0065dd0 tx=0x7f63f0006690).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:06.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018390 secure :-1 s=READY pgs=5844 cs=0 l=1 rev1=1 rx=0x7f63e0065dd0 tx=0x7f63f0006690).stop 2021-03-09T09:24:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0018390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:06.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0018390 unknown :-1 s=CLOSED pgs=5844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:06.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:24:06.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:06.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:06.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:06.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:06.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:06.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:06.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:06.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012710 con 0x7f63e00070a0 2021-03-09T09:24:06.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d680 con 0x7f6400145bd0 2021-03-09T09:24:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.477+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.477+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024cc0 con 0x7f63e005c7c0 2021-03-09T09:24:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.477+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.477+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:06.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.477+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:06.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.477+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:06.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.477+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:06.487 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:06.478+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15787 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.478+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:06.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.478+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:24:06.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.478+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:06.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.478+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:06.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.479+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:06.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.479+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:06.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.479+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:06.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:06.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:06.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:06.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:06.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:06.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:24:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:24:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002b9d0 con 0x7f63e005c7c0 2021-03-09T09:24:06.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:06.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:06.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:24:06.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:24:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.480+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.481+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:24:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.481+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.481+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5865 cs=0 l=1 rev1=1 rx=0x7f63f007fe40 tx=0x7f63f007feb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e005c7c0 2021-03-09T09:24:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.482+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.482+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.482+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:06.497 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:06.497 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:06.497 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:06.497 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:06.498 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:06.498 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:06.505 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:06.505 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:06.505 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:06.506 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:06.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e005c7c0 2021-03-09T09:24:06.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.484+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:06.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.484+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:24:06.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f006e520 con 0x7f63e005c7c0 2021-03-09T09:24:06.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:06.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:06.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:07.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:07.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:07.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:08.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:08.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:08.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:08.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:09.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:09.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:09.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:09.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:09.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:09.753+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:10.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:10.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:10.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:10.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:11.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5865 cs=0 l=1 rev1=1 rx=0x7f63f007fe40 tx=0x7f63f007feb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:11.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5865 cs=0 l=1 rev1=1 rx=0x7f63f007fe40 tx=0x7f63f007feb0).stop 2021-03-09T09:24:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:11.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:11.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:11.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=5865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:24:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.479+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:11.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:11.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00655f0 con 0x7f63e00070a0 2021-03-09T09:24:11.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:11.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:24:11.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:11.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010350 con 0x7f63e0064730 2021-03-09T09:24:11.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:11.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:11.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:11.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.480+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:24:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:11.495 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15827 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:11.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:11.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:11.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:11.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:11.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:11.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.481+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:11.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:24:11.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:24:11.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T09:24:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017c30 con 0x7f63e0064730 2021-03-09T09:24:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:24:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d9e0 secure :-1 s=READY pgs=5876 cs=0 l=1 rev1=1 rx=0x7f63f4039050 tx=0x7f63f4003b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:24:11.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.482+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:11.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63e0064730 2021-03-09T09:24:11.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.484+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:11.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.484+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.484+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:11.506 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:11.506 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:11.506 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:11.507 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:11.507 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:11.507 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:11.508 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:11.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40183c0 con 0x7f63e0064730 2021-03-09T09:24:11.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.484+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:11.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.484+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:24:11.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400f7d0 con 0x7f63e0064730 2021-03-09T09:24:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:11.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:12.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:12.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:12.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:12.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:12.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:12.611+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:13.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:13.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:13.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:13.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:14.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:14.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:14.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:14.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:14.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:14.754+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:15.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:15.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:15.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:15.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:16.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.481+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:16.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.481+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:16.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d9e0 secure :-1 s=READY pgs=5876 cs=0 l=1 rev1=1 rx=0x7f63f4039050 tx=0x7f63f4003b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:16.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d9e0 secure :-1 s=READY pgs=5876 cs=0 l=1 rev1=1 rx=0x7f63f4039050 tx=0x7f63f4003b40).stop 2021-03-09T09:24:16.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:16.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:16.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001d9e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:16.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001d9e0 unknown :-1 s=CLOSED pgs=5876 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:16.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:24:16.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:16.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:16.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:16.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:16.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:16.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:16.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:16.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067ab0 con 0x7f63e00070a0 2021-03-09T09:24:16.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:16.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d6f0 con 0x7f6400145bd0 2021-03-09T09:24:16.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:16.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012710 con 0x7f63e005c7c0 2021-03-09T09:24:16.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:16.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:16.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:16.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:16.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:16.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:16.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:24:16.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:16.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.482+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:16.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.483+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:16.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.483+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:16.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.483+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:16.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.483+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:16.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.483+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:16.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:16.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:16.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:24:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:24:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:16.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f470 con 0x7f63e005c7c0 2021-03-09T09:24:16.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:16.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:16.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:24:16.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:24:16.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:16.497 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:16.484+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15864 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:16.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.485+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5887 cs=0 l=1 rev1=1 rx=0x7f63f4062be0 tx=0x7f63f4062cc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:16.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.485+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:24:16.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.485+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:16.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.486+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400fd50 con 0x7f63e005c7c0 2021-03-09T09:24:16.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.486+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:16.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.486+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:16.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.486+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:16.500 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:16.500 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:16.500 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:16.500 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:16.500 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:16.501 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:16.501 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:16.501 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:16.501 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:16.502 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:16.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63e005c7c0 2021-03-09T09:24:16.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.487+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:16.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400fd50 con 0x7f63e005c7c0 2021-03-09T09:24:16.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.487+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:24:16.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:16.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:16.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:17.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:17.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:17.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:17.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:18.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:18.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:18.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:19.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:19.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:19.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:19.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:19.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:19.755+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:20.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:20.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:20.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:20.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:24:21.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5887 cs=0 l=1 rev1=1 rx=0x7f63f4062be0 tx=0x7f63f4062cc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5887 cs=0 l=1 rev1=1 rx=0x7f63f4062be0 tx=0x7f63f4062cc0).stop 2021-03-09T09:24:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=5887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:24:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:21.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:21.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:21.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:21.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:21.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:21.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3b0 con 0x7f63e00070a0 2021-03-09T09:24:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078110 con 0x7f6400145bd0 2021-03-09T09:24:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:24:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.484+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:21.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:21.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:21.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:21.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:21.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:24:21.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:21.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:21.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:21.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:21.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.485+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:21.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:21.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:21.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:21.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:21.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:21.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:24:21.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:24:21.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:21.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:21.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b7d0 con 0x7f63e0064730 2021-03-09T09:24:21.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:21.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:21.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002b9d0 con 0x7f63e0064730 2021-03-09T09:24:21.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:24:21.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:21.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:24:21.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:21.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ee30 secure :-1 s=READY pgs=5898 cs=0 l=1 rev1=1 rx=0x7f63e0012150 tx=0x7f63f000eef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:21.500 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:21.486+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15904 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:21.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.488+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030e80 con 0x7f63e0064730 2021-03-09T09:24:21.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.488+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:21.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.488+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:21.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.488+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:21.501 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:21.502 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:21.502 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:21.502 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:21.502 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:21.503 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:21.503 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:21.503 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:21.503 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:21.503 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.488+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030e80 con 0x7f63e0064730 2021-03-09T09:24:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.488+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.488+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:24:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.489+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63e0064730 2021-03-09T09:24:21.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:21.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:21.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:22.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:22.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:22.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:22.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:22.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:22.612+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:23.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:23.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:23.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:23.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:24.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:24.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:24.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:24.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:24.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:24.756+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:25.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:25.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:25.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:25.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:26.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:26.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:26.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ee30 secure :-1 s=READY pgs=5898 cs=0 l=1 rev1=1 rx=0x7f63e0012150 tx=0x7f63f000eef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:26.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ee30 secure :-1 s=READY pgs=5898 cs=0 l=1 rev1=1 rx=0x7f63e0012150 tx=0x7f63f000eef0).stop 2021-03-09T09:24:26.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:26.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:26.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001ee30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:26.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001ee30 unknown :-1 s=CLOSED pgs=5898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:26.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:24:26.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:26.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:26.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:26.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:26.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:26.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:26.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:26.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d6f0 con 0x7f63e00070a0 2021-03-09T09:24:26.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:26.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012710 con 0x7f6400145bd0 2021-03-09T09:24:26.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:26.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00655f0 con 0x7f63e005c7c0 2021-03-09T09:24:26.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:26.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:26.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:26.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.488+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:26.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.488+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:24:26.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.488+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:26.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.488+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:26.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.488+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:26.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.488+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:26.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.488+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:26.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.488+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:26.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.488+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:26.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:26.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:26.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:26.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:26.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:26.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:24:26.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:24:26.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:26.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:26.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:24:26.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:26.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:26.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f470 con 0x7f63e005c7c0 2021-03-09T09:24:26.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:24:26.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:26.503 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:26.489+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15941 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:26.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.490+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5914 cs=0 l=1 rev1=1 rx=0x7f63f406b040 tx=0x7f63f40632c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:26.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.490+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:24:26.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.490+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:26.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40035e0 con 0x7f63e005c7c0 2021-03-09T09:24:26.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.491+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:26.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.491+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:26.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.491+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:26.506 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:26.506 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:26.506 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:26.506 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:26.508 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:26.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40035e0 con 0x7f63e005c7c0 2021-03-09T09:24:26.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.491+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:26.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.492+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:24:26.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e005c7c0 2021-03-09T09:24:26.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:26.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:26.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:27.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:27.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:27.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:27.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:28.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:28.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:28.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:28.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:29.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:29.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:29.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:29.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:29.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:29.757+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:30.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:30.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:30.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:31.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:31.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:31.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5914 cs=0 l=1 rev1=1 rx=0x7f63f406b040 tx=0x7f63f40632c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:31.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5914 cs=0 l=1 rev1=1 rx=0x7f63f406b040 tx=0x7f63f40632c0).stop 2021-03-09T09:24:31.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:31.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:31.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:31.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=5914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:31.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:24:31.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:31.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:31.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:31.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:31.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:31.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:31.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:31.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:24:31.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:31.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068a20 con 0x7f6400145bd0 2021-03-09T09:24:31.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:31.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a750 con 0x7f63e0064730 2021-03-09T09:24:31.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:31.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:31.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:31.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:31.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:31.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.491+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:31.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.491+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:24:31.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.491+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:31.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.491+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:31.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.491+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:31.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.491+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:31.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.491+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:31.500 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:31.491+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 15981 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:31.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:31.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:31.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:31.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:31.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:31.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:24:31.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:24:31.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:31.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:31.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018120 con 0x7f63e0064730 2021-03-09T09:24:31.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:31.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:31.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0064730 2021-03-09T09:24:31.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:24:31.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.492+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:31.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.493+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 secure :-1 s=READY pgs=5927 cs=0 l=1 rev1=1 rx=0x7f63f404a010 tx=0x7f63f4053d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:31.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.493+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:24:31.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.493+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:31.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072b10 con 0x7f63e0064730 2021-03-09T09:24:31.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.495+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:31.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.495+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:31.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.495+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:31.507 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:31.507 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:31.507 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:31.508 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:31.508 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:31.508 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:31.508 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:31.508 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:31.509 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:31.509 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:31.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072c70 con 0x7f63e0064730 2021-03-09T09:24:31.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.495+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:31.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.495+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:24:31.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f401b2e0 con 0x7f63e0064730 2021-03-09T09:24:31.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:31.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:31.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:32.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:32.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:32.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:32.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:32.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:32.614+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:33.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:33.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:33.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:33.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:34.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:34.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:34.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:34.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:34.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:34.758+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:35.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:35.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:35.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:35.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:36.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:36.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:36.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 secure :-1 s=READY pgs=5927 cs=0 l=1 rev1=1 rx=0x7f63f404a010 tx=0x7f63f4053d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:36.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 secure :-1 s=READY pgs=5927 cs=0 l=1 rev1=1 rx=0x7f63f404a010 tx=0x7f63f4053d90).stop 2021-03-09T09:24:36.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:36.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:36.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:36.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 unknown :-1 s=CLOSED pgs=5927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:36.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:24:36.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:36.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:36.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:36.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:36.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:36.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:36.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:36.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b4e0 con 0x7f63e00070a0 2021-03-09T09:24:36.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:36.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dc80 con 0x7f6400145bd0 2021-03-09T09:24:36.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:36.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f0c0 con 0x7f63e005c7c0 2021-03-09T09:24:36.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:36.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:36.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:36.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:36.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:24:36.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:36.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.492+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:36.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:36.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:36.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:36.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:24:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:24:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63e005c7c0 2021-03-09T09:24:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:36.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:36.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024540 con 0x7f63e005c7c0 2021-03-09T09:24:36.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:24:36.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.493+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:36.507 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:36.494+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16018 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:36.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.494+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:24:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.494+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5938 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc012ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.494+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63e005c7c0 2021-03-09T09:24:36.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.496+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.496+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.496+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:36.510 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:36.511 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:36.512 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:36.512 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:36.512 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:36.512 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:36.512 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:36.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:24:36.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.496+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:36.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.496+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:24:36.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.497+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63e005c7c0 2021-03-09T09:24:36.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:36.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:36.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:37.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:37.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:37.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:37.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:38.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:38.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:38.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:38.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:39.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:39.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:39.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:39.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:39.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:39.759+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:40.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:40.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:40.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:40.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:41.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:41.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:41.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5938 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc012ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:41.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5938 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc012ed0).stop 2021-03-09T09:24:41.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:41.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:41.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:41.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=5938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:41.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:24:41.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:41.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:41.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:41.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:41.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:41.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:41.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:41.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d6f0 con 0x7f63e00070a0 2021-03-09T09:24:41.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:41.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00655f0 con 0x7f6400145bd0 2021-03-09T09:24:41.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:41.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:24:41.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:41.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:41.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:41.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:41.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.495+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:41.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.496+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:41.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.496+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:24:41.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.496+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:41.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.496+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:41.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.496+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:41.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.496+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:41.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.496+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:41.506 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:41.496+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16058 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:41.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:41.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:41.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:41.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:41.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:41.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:24:41.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:24:41.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:41.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:41.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029800 con 0x7f63e0064730 2021-03-09T09:24:41.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:41.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:41.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018120 con 0x7f63e0064730 2021-03-09T09:24:41.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:24:41.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:41.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 secure :-1 s=READY pgs=5958 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4017e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:41.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:24:41.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.497+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:41.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0064730 2021-03-09T09:24:41.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.499+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:41.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.499+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:41.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.499+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:41.512 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:41.512 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:41.512 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:41.512 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:41.513 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:41.513 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:41.513 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:41.513 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:41.513 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:41.514 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:41.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0064730 2021-03-09T09:24:41.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.500+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:41.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.500+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:24:41.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4023dd0 con 0x7f63e0064730 2021-03-09T09:24:41.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:41.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:41.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:42.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:42.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:42.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:42.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:42.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:42.615+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:43.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:43.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:43.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:43.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:44.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:44.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:44.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:44.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:44.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:44.760+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:45.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:45.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:46.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:46.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:46.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 secure :-1 s=READY pgs=5958 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4017e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:46.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 secure :-1 s=READY pgs=5958 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4017e60).stop 2021-03-09T09:24:46.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:46.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:46.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:46.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 unknown :-1 s=CLOSED pgs=5958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:46.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:24:46.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:46.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:46.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:46.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:46.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:46.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:46.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:46.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a9e0 con 0x7f63e00070a0 2021-03-09T09:24:46.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:46.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077fa0 con 0x7f6400145bd0 2021-03-09T09:24:46.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:46.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027c30 con 0x7f63e005c7c0 2021-03-09T09:24:46.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:46.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:46.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:46.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:46.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.497+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:46.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:46.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:24:46.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:46.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:46.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:46.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:46.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:46.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:46.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:46.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:46.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:46.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:46.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:24:46.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:24:46.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:46.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:46.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63e005c7c0 2021-03-09T09:24:46.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:46.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:46.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b7d0 con 0x7f63e005c7c0 2021-03-09T09:24:46.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:24:46.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.498+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:46.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.499+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:24:46.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.499+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:46.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.499+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5966 cs=0 l=1 rev1=1 rx=0x7f63f002a040 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:46.517 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:46.499+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16095 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:46.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e005c7c0 2021-03-09T09:24:46.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.500+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:46.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.500+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:46.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.500+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:46.519 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:46.519 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:46.519 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:46.520 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:46.520 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:46.520 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:46.521 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:46.521 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:46.521 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:46.522 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:46.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e005c7c0 2021-03-09T09:24:46.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.500+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:46.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.500+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:24:46.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.501+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0003b10 con 0x7f63e005c7c0 2021-03-09T09:24:46.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:46.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:46.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:47.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:47.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:47.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:47.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:48.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:48.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:48.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:48.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:49.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:49.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:49.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:49.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:49.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:49.761+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:50.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:50.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:50.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:50.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:51.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:51.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:51.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5966 cs=0 l=1 rev1=1 rx=0x7f63f002a040 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:51.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5966 cs=0 l=1 rev1=1 rx=0x7f63f002a040 tx=0x7f63f0013b80).stop 2021-03-09T09:24:51.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:51.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:51.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:51.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=5966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:51.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:24:51.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:51.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:51.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:51.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:51.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:51.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:51.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:51.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021da0 con 0x7f63e00070a0 2021-03-09T09:24:51.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:51.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00655f0 con 0x7f6400145bd0 2021-03-09T09:24:51.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:51.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63e0064730 2021-03-09T09:24:51.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:51.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:51.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:51.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:51.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.500+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:51.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.501+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:51.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.501+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:24:51.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.501+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:51.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.501+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:51.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:51.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:51.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:51.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:51.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:51.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:51.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:51.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:51.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:24:51.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:24:51.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:51.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:51.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0064730 2021-03-09T09:24:51.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:51.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:51.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63e0064730 2021-03-09T09:24:51.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:24:51.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:51.515 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16135 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:51.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.502+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 secure :-1 s=READY pgs=5984 cs=0 l=1 rev1=1 rx=0x7f63fc012ff0 tx=0x7f63fc03c3c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:51.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.503+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:24:51.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.503+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:51.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.504+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63e0064730 2021-03-09T09:24:51.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.504+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:51.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.504+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:51.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.504+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:51.518 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:51.518 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:51.518 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:51.518 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:51.519 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:51.519 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:51.519 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:51.519 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:51.520 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:51.520 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:51.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63e0064730 2021-03-09T09:24:51.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.506+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:51.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.506+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:24:51.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.508+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e0064730 2021-03-09T09:24:51.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:51.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:51.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:52.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:52.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:52.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:52.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:52.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:52.616+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:53.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:53.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:53.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:53.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:54.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:54.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:54.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:54.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:54.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:54.762+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:24:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:55.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:55.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:55.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:56.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:24:56.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:24:56.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 secure :-1 s=READY pgs=5984 cs=0 l=1 rev1=1 rx=0x7f63fc012ff0 tx=0x7f63fc03c3c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:24:56.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 secure :-1 s=READY pgs=5984 cs=0 l=1 rev1=1 rx=0x7f63fc012ff0 tx=0x7f63fc03c3c0).stop 2021-03-09T09:24:56.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:56.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:24:56.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:24:56.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e002da40 unknown :-1 s=CLOSED pgs=5984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:24:56.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:24:56.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:56.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:24:56.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:56.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:24:56.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:24:56.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:24:56.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:56.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e00070a0 2021-03-09T09:24:56.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:56.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012710 con 0x7f6400145bd0 2021-03-09T09:24:56.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:24:56.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022650 con 0x7f63e005c7c0 2021-03-09T09:24:56.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:24:56.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:24:56.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:56.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:24:56.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:24:56.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:24:56.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:24:56.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:24:56.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:24:56.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:24:56.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:24:56.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.503+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:24:56.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:24:56.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:56.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:56.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:24:56.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:24:56.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:24:56.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:24:56.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:24:56.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:56.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63e005c7c0 2021-03-09T09:24:56.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:24:56.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:24:56.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63e005c7c0 2021-03-09T09:24:56.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:24:56.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:56.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:24:56.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5994 cs=0 l=1 rev1=1 rx=0x7f63f001c630 tx=0x7f63f0006ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:24:56.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.504+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:24:56.519 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:24:56.505+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16172 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:24:56.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.505+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001b040 con 0x7f63e005c7c0 2021-03-09T09:24:56.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.505+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:24:56.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.505+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:24:56.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.505+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:24:56.520 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:24:56.521 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:24:56.521 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:56.521 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:24:56.521 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:24:56.522 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:24:56.522 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:24:56.522 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:24:56.522 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:24:56.522 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:24:56.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.505+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001b1a0 con 0x7f63e005c7c0 2021-03-09T09:24:56.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.505+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:24:56.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.505+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:24:56.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001c290 con 0x7f63e005c7c0 2021-03-09T09:24:56.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:56.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:56.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:57.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:57.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:57.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:57.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:58.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:58.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:58.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:58.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:59.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:59.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:24:59.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:59.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:24:59.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:24:59.763+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:00.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:00.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:00.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:00.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:01.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:25:01.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:01.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5994 cs=0 l=1 rev1=1 rx=0x7f63f001c630 tx=0x7f63f0006ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:01.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=5994 cs=0 l=1 rev1=1 rx=0x7f63f001c630 tx=0x7f63f0006ca0).stop 2021-03-09T09:25:01.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:01.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:01.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:01.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=5994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:01.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:25:01.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:01.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:01.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:01.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:01.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:01.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:01.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:01.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:25:01.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:01.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00655f0 con 0x7f6400145bd0 2021-03-09T09:25:01.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:01.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cc00 con 0x7f63e0064730 2021-03-09T09:25:01.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:01.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:01.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:01.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:01.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:01.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:01.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:25:01.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:01.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:01.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:01.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:01.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.506+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:01.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:01.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:01.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:01.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:01.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:01.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:25:01.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:25:01.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:01.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:01.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0077f50 con 0x7f63e0064730 2021-03-09T09:25:01.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:01.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:01.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0064730 2021-03-09T09:25:01.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:25:01.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:01.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:25:01.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016bd0 secure :-1 s=READY pgs=6007 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc032360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:01.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:01.520 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:01.507+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16212 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:01.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03c170 con 0x7f63e0064730 2021-03-09T09:25:01.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.509+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:01.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.509+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:01.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.509+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:01.522 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:01.522 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:01.522 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:01.522 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:01.523 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:01.523 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:01.523 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:01.523 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:01.524 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:01.524 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:01.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e0064730 2021-03-09T09:25:01.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.509+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:01.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.509+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:25:01.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.510+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e0064730 2021-03-09T09:25:01.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:01.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:01.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:02.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:02.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:02.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:02.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:02.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:02.618+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:03.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:03.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:03.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:03.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:04.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:04.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:04.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:04.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:04.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:04.764+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:05.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:05.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:05.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:05.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:06.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0016bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:06.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0016bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:06.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016bd0 secure :-1 s=READY pgs=6007 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc032360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:06.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016bd0 secure :-1 s=READY pgs=6007 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc032360).stop 2021-03-09T09:25:06.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:06.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:06.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0016bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:06.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016bd0 unknown :-1 s=CLOSED pgs=6007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:06.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:25:06.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:06.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:06.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:06.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:06.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:06.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:06.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:06.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d6f0 con 0x7f63e00070a0 2021-03-09T09:25:06.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:06.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00655f0 con 0x7f6400145bd0 2021-03-09T09:25:06.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:06.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e9d0 con 0x7f63e005c7c0 2021-03-09T09:25:06.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:06.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:06.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.508+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:06.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.509+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:06.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.509+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:25:06.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.509+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:06.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.509+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:06.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.509+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:06.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.509+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:06.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.509+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:06.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:06.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:06.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:06.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:06.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:06.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:25:06.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:25:06.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:06.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:06.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010300 con 0x7f63e005c7c0 2021-03-09T09:25:06.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:06.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:06.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0077f50 con 0x7f63e005c7c0 2021-03-09T09:25:06.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:25:06.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:06.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6021 cs=0 l=1 rev1=1 rx=0x7f63fc04d040 tx=0x7f63fc032230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:06.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:25:06.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:06.525 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:06.510+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16249 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.512+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066c60 con 0x7f63e005c7c0 2021-03-09T09:25:06.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.512+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:06.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.512+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:06.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.512+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:06.526 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:06.528 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:06.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.512+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0710f0 con 0x7f63e005c7c0 2021-03-09T09:25:06.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.512+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:06.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.512+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:25:06.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.513+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63e005c7c0 2021-03-09T09:25:06.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:06.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:06.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:07.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:07.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:07.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:08.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:08.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:08.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:08.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:09.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:09.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:09.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:09.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:09.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:09.765+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:10.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:10.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:10.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:10.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:11.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:25:11.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:11.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6021 cs=0 l=1 rev1=1 rx=0x7f63fc04d040 tx=0x7f63fc032230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:11.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6021 cs=0 l=1 rev1=1 rx=0x7f63fc04d040 tx=0x7f63fc032230).stop 2021-03-09T09:25:11.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:11.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:11.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:11.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:11.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:25:11.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:11.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:11.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:11.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:11.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:11.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:11.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:11.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a750 con 0x7f63e00070a0 2021-03-09T09:25:11.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:11.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013250 con 0x7f6400145bd0 2021-03-09T09:25:11.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:11.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f6d0 con 0x7f63e0064730 2021-03-09T09:25:11.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:11.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:25:11.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:11.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:11.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:11.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:11.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.511+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:11.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:11.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:11.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:11.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:25:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:25:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:11.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b490 con 0x7f63e0064730 2021-03-09T09:25:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010300 con 0x7f63e0064730 2021-03-09T09:25:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:25:11.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.512+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:11.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.513+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016670 secure :-1 s=READY pgs=6035 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc033820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:11.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.513+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:25:11.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.513+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:11.527 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:11.513+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16289 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:11.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0710f0 con 0x7f63e0064730 2021-03-09T09:25:11.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.514+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:11.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.514+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:11.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.514+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:11.528 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:11.529 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:11.529 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:11.529 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:11.529 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:11.529 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:11.530 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:11.530 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:11.530 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:11.530 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:11.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03c170 con 0x7f63e0064730 2021-03-09T09:25:11.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.514+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:11.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.514+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:25:11.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.516+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63e0064730 2021-03-09T09:25:11.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:11.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:11.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:12.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:12.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:12.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:12.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:12.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:12.619+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:13.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:13.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:13.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:13.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:14.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:14.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:14.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:14.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:14.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:14.766+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:15.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:15.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:15.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:15.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:16.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0016670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:16.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0016670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:16.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016670 secure :-1 s=READY pgs=6035 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc033820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:16.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016670 secure :-1 s=READY pgs=6035 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc033820).stop 2021-03-09T09:25:16.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:16.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:16.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0016670 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:16.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0016670 unknown :-1 s=CLOSED pgs=6035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:16.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:25:16.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:16.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:16.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:16.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:16.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:16.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:16.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:16.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ca30 con 0x7f63e00070a0 2021-03-09T09:25:16.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:16.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:25:16.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:16.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00655f0 con 0x7f63e005c7c0 2021-03-09T09:25:16.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:16.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:16.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:16.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:16.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.514+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:16.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.515+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:16.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.515+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:25:16.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.515+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:16.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.515+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:16.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:16.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:16.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:16.532 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16326 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:16.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:16.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:16.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:16.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:16.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:16.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:25:16.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:25:16.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:16.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:16.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b300 con 0x7f63e005c7c0 2021-03-09T09:25:16.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:16.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:16.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b490 con 0x7f63e005c7c0 2021-03-09T09:25:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:25:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.516+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.517+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6054 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc032440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.517+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:25:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.517+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.518+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03c170 con 0x7f63e005c7c0 2021-03-09T09:25:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.518+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.518+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.518+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:16.667 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:16.667 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:16.667 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:16.667 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:16.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63e005c7c0 2021-03-09T09:25:16.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.520+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:16.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.520+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:25:16.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.521+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc052ce0 con 0x7f63e005c7c0 2021-03-09T09:25:16.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:16.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:16.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:17.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:17.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:17.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:17.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:18.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:18.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:18.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:18.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:19.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:19.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:19.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:19.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:19.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:19.767+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:20.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:20.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:20.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:20.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:21.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:21.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:21.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6054 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc032440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:21.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6054 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc032440).stop 2021-03-09T09:25:21.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:21.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:21.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:21.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:21.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:25:21.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:21.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:21.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:21.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:21.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:21.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:21.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:21.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ced0 con 0x7f63e00070a0 2021-03-09T09:25:21.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:21.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019ac0 con 0x7f6400145bd0 2021-03-09T09:25:21.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:21.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d6f0 con 0x7f63e0064730 2021-03-09T09:25:21.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:21.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:21.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:21.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:21.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:21.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:21.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:25:21.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:21.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.517+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:21.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:21.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:21.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:21.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:21.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:21.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:21.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:21.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:21.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:25:21.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:25:21.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:21.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:21.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d60 con 0x7f63e0064730 2021-03-09T09:25:21.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:21.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:21.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029800 con 0x7f63e0064730 2021-03-09T09:25:21.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:25:21.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.518+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:21.532 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:21.519+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16366 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:21.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.519+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:25:21.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.519+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=6065 cs=0 l=1 rev1=1 rx=0x7f63f405b7b0 tx=0x7f63f40051f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:21.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.519+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:21.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e0064730 2021-03-09T09:25:21.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.522+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:21.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.522+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:21.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.522+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:21.534 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:21.535 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:21.535 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:21.535 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:21.535 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:21.535 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:21.536 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:21.536 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:21.536 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:21.536 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:21.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0064730 2021-03-09T09:25:21.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.522+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:21.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.522+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:25:21.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e0064730 2021-03-09T09:25:21.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:21.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:21.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:22.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:22.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:22.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:22.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:22.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:22.621+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:23.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:23.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:23.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:23.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:24.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:24.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:24.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:24.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:24.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:24.768+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:25.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:25.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:25.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:25.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:26.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:26.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:26.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=6065 cs=0 l=1 rev1=1 rx=0x7f63f405b7b0 tx=0x7f63f40051f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:26.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=6065 cs=0 l=1 rev1=1 rx=0x7f63f405b7b0 tx=0x7f63f40051f0).stop 2021-03-09T09:25:26.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=CLOSED pgs=6065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:25:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:26.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:26.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:26.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:26.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:26.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:26.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:26.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de20 con 0x7f63e00070a0 2021-03-09T09:25:26.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:26.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011df0 con 0x7f6400145bd0 2021-03-09T09:25:26.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:26.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:25:26.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:26.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:26.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.519+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:26.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.520+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:26.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.520+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:26.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.520+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:26.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.520+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:25:26.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.520+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:26.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.520+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:26.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.520+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:26.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.520+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:26.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.520+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:26.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:26.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:26.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:26.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:26.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:26.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:25:26.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:25:26.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:26.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:26.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b490 con 0x7f63e005c7c0 2021-03-09T09:25:26.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:26.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:26.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63e005c7c0 2021-03-09T09:25:26.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:25:26.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:26.686 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16403 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:26.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:25:26.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6077 cs=0 l=1 rev1=1 rx=0x7f63e0012960 tx=0x7f63f0013b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:26.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.521+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:26.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001b040 con 0x7f63e005c7c0 2021-03-09T09:25:26.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.524+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:26.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.524+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:26.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.524+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:26.689 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:26.689 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:26.689 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:26.690 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:26.690 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:26.690 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:26.690 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:26.690 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:26.691 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:26.691 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:26.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001b1a0 con 0x7f63e005c7c0 2021-03-09T09:25:26.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.524+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:26.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00036b0 con 0x7f63e005c7c0 2021-03-09T09:25:26.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.524+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:25:26.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:26.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:26.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:27.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:27.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:27.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:27.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:28.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:28.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:28.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:28.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:29.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:29.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:29.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:29.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:29.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:29.769+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:30.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:30.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:30.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:30.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:31.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:31.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:31.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6077 cs=0 l=1 rev1=1 rx=0x7f63e0012960 tx=0x7f63f0013b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6077 cs=0 l=1 rev1=1 rx=0x7f63e0012960 tx=0x7f63f0013b10).stop 2021-03-09T09:25:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6077 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:25:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:31.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:31.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:31.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:31.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:31.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a300 con 0x7f63e00070a0 2021-03-09T09:25:31.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:31.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018c70 con 0x7f6400145bd0 2021-03-09T09:25:31.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:31.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b820 con 0x7f63e0064730 2021-03-09T09:25:31.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:31.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:31.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:31.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:25:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:31.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:31.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:31.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.522+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:31.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:31.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:31.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:31.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:31.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:31.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:25:31.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:25:31.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:31.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:31.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63e0064730 2021-03-09T09:25:31.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:31.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:31.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002d60 con 0x7f63e0064730 2021-03-09T09:25:31.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:25:31.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.523+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:31.536 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:31.524+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16443 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:31.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.524+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=6091 cs=0 l=1 rev1=1 rx=0x7f63f4063000 tx=0x7f63f4003650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:31.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.524+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:25:31.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.524+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:31.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e0064730 2021-03-09T09:25:31.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.525+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:31.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.525+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:31.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.525+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:31.539 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:31.539 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:31.539 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:31.539 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:31.540 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:31.540 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:31.540 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:31.540 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:31.540 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:31.541 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:31.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63e0064730 2021-03-09T09:25:31.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.525+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:31.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.525+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:25:31.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.527+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4009830 con 0x7f63e0064730 2021-03-09T09:25:31.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:31.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:31.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:32.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:32.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:32.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:32.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:32.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:32.622+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:33.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:33.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:33.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:33.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:34.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:34.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:34.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:34.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:34.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:34.770+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:35.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:35.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:35.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:35.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=6091 cs=0 l=1 rev1=1 rx=0x7f63f4063000 tx=0x7f63f4003650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 secure :-1 s=READY pgs=6091 cs=0 l=1 rev1=1 rx=0x7f63f4063000 tx=0x7f63f4003650).stop 2021-03-09T09:25:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0023930 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0023930 unknown :-1 s=CLOSED pgs=6091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:36.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:25:36.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:36.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:36.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:36.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:36.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:36.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:36.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:36.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027c30 con 0x7f63e00070a0 2021-03-09T09:25:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dc80 con 0x7f6400145bd0 2021-03-09T09:25:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012710 con 0x7f63e005c7c0 2021-03-09T09:25:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:36.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:36.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:36.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:36.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.525+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:36.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:36.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:25:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:36.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:36.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:36.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:36.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0008810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:36.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:25:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:25:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:36.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:25:36.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:36.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:36.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b300 con 0x7f63e005c7c0 2021-03-09T09:25:36.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:25:36.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.526+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:36.540 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:36.527+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16480 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:36.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.527+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6105 cs=0 l=1 rev1=1 rx=0x7f63fc01b210 tx=0x7f63fc04d580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:36.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.527+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:25:36.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.527+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:36.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00f140 con 0x7f63e005c7c0 2021-03-09T09:25:36.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.529+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:36.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.529+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:36.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.529+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:36.543 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:36.543 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:36.543 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:36.543 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:36.544 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:36.544 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:36.544 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:36.544 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:36.545 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:36.545 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:36.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e005c7c0 2021-03-09T09:25:36.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.529+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:36.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.529+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:25:36.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.530+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:25:36.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:36.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:36.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:37.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:37.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:37.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:37.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:38.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:38.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:38.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:38.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:39.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:39.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:39.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:39.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:39.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:39.771+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:40.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:40.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:40.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:40.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:41.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:25:41.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:41.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6105 cs=0 l=1 rev1=1 rx=0x7f63fc01b210 tx=0x7f63fc04d580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:41.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6105 cs=0 l=1 rev1=1 rx=0x7f63fc01b210 tx=0x7f63fc04d580).stop 2021-03-09T09:25:41.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:41.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:41.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:41.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:41.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:25:41.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0013480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:41.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:41.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d6f0 con 0x7f63e00070a0 2021-03-09T09:25:41.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:41.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de20 con 0x7f6400145bd0 2021-03-09T09:25:41.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f1a0 con 0x7f63e0064730 2021-03-09T09:25:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:41.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:41.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0013480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:41.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:25:41.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.528+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:41.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:41.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:41.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:41.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:25:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:25:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:41.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T09:25:41.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:41.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:41.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63e0064730 2021-03-09T09:25:41.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:25:41.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:41.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0013480 secure :-1 s=READY pgs=6119 cs=0 l=1 rev1=1 rx=0x7f63f404be00 tx=0x7f63f404bee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:41.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:25:41.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.529+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:41.544 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:41.530+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16519 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:41.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072d40 con 0x7f63e0064730 2021-03-09T09:25:41.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.531+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:41.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.531+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:41.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.531+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:41.549 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:41.549 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:41.549 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:41.549 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:41.550 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:41.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.532+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072aa0 con 0x7f63e0064730 2021-03-09T09:25:41.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.532+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:41.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.532+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:25:41.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.533+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4072d40 con 0x7f63e0064730 2021-03-09T09:25:41.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:41.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:41.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:42.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:42.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:42.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:42.623+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:43.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:43.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:43.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:43.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:44.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:44.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:44.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:44.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:44.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:44.772+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:45.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:45.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:45.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:45.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:46.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0013480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:25:46.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0013480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:46.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0013480 secure :-1 s=READY pgs=6119 cs=0 l=1 rev1=1 rx=0x7f63f404be00 tx=0x7f63f404bee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:46.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0013480 secure :-1 s=READY pgs=6119 cs=0 l=1 rev1=1 rx=0x7f63f404be00 tx=0x7f63f404bee0).stop 2021-03-09T09:25:46.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:46.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:46.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0013480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:46.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0013480 unknown :-1 s=CLOSED pgs=6119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:46.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:25:46.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:46.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:46.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:46.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:46.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:46.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:46.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:46.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d8f0 con 0x7f63e00070a0 2021-03-09T09:25:46.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:46.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:25:46.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:46.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de20 con 0x7f63e005c7c0 2021-03-09T09:25:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:46.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.531+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.532+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:46.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.532+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.532+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.532+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:25:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.532+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.532+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:46.541 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16557 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:46.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:46.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:46.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:46.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:25:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:25:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f470 con 0x7f63e005c7c0 2021-03-09T09:25:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.533+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b490 con 0x7f63e005c7c0 2021-03-09T09:25:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.534+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:25:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.534+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.534+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:25:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.534+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.534+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6134 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007ffd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.535+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001b250 con 0x7f63e005c7c0 2021-03-09T09:25:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.535+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.535+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.535+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:46.549 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:46.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.535+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0003b10 con 0x7f63e005c7c0 2021-03-09T09:25:46.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.535+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:46.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.535+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:25:46.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.536+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0003c80 con 0x7f63e005c7c0 2021-03-09T09:25:46.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:46.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:46.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:47.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:47.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:47.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:47.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:48.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:48.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:48.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:48.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:49.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:49.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:49.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:49.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:49.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:49.773+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:50.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:50.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:50.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:50.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:51.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:25:51.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:51.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6134 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007ffd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:51.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6134 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007ffd0).stop 2021-03-09T09:25:51.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:51.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:51.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:51.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:51.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:25:51.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:51.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:51.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:51.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:51.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00681c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:51.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:51.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:51.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f1a0 con 0x7f63e00070a0 2021-03-09T09:25:51.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:51.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e230 con 0x7f6400145bd0 2021-03-09T09:25:51.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:51.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027c30 con 0x7f63e0064730 2021-03-09T09:25:51.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:51.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:51.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:51.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:51.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.534+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:51.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.535+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00681c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:51.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.535+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:25:51.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.535+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:51.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.535+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:51.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.535+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:51.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.535+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:51.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.535+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:51.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:51.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:51.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:51.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:51.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:51.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:25:51.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:25:51.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:51.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:51.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d4f0 con 0x7f63e0064730 2021-03-09T09:25:51.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:51.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:51.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f470 con 0x7f63e0064730 2021-03-09T09:25:51.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:25:51.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:51.549 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:51.536+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16595 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:51.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.537+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00681c0 secure :-1 s=READY pgs=6149 cs=0 l=1 rev1=1 rx=0x7f63f000e020 tx=0x7f63f0003e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:51.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.537+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:25:51.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.537+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:51.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055d70 con 0x7f63e0064730 2021-03-09T09:25:51.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.538+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:51.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.538+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:51.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.538+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:51.553 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:51.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055d70 con 0x7f63e0064730 2021-03-09T09:25:51.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.539+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:51.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.539+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:25:51.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.540+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00554a0 con 0x7f63e0064730 2021-03-09T09:25:51.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:51.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:51.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:52.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:52.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:52.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:52.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:52.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:52.625+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:53.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:53.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:53.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:53.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:54.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:54.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:54.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:54.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:54.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:54.774+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:25:55.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:55.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:55.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:55.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:56.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00681c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:25:56.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00681c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:25:56.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00681c0 secure :-1 s=READY pgs=6149 cs=0 l=1 rev1=1 rx=0x7f63f000e020 tx=0x7f63f0003e50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:25:56.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00681c0 secure :-1 s=READY pgs=6149 cs=0 l=1 rev1=1 rx=0x7f63f000e020 tx=0x7f63f0003e50).stop 2021-03-09T09:25:56.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:56.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:25:56.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00681c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:25:56.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00681c0 unknown :-1 s=CLOSED pgs=6149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:25:56.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:25:56.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.536+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:56.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:25:56.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:56.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:25:56.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:25:56.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:25:56.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:56.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f63e00070a0 2021-03-09T09:25:56.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:56.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d6f0 con 0x7f6400145bd0 2021-03-09T09:25:56.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:25:56.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:25:56.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:25:56.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:25:56.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:56.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:25:56.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.537+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:25:56.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.538+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:25:56.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.538+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:25:56.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.538+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:25:56.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.538+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:25:56.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.538+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:25:56.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.538+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:25:56.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.538+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:25:56.546 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16644 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:25:56.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:25:56.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:56.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:56.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:25:56.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:25:56.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:25:56.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:25:56.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:25:56.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:56.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63e005c7c0 2021-03-09T09:25:56.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:25:56.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:25:56.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d4f0 con 0x7f63e005c7c0 2021-03-09T09:25:56.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:25:56.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:56.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6162 cs=0 l=1 rev1=1 rx=0x7f63f0080040 tx=0x7f63f0030ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:25:56.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:25:56.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.539+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:25:56.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.540+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001ceb0 con 0x7f63e005c7c0 2021-03-09T09:25:56.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.540+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:25:56.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.540+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:25:56.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.540+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:25:56.553 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:25:56.553 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:25:56.553 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:56.553 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:25:56.553 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:25:56.554 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:25:56.554 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:25:56.554 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:25:56.554 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:25:56.554 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:25:56.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.540+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055ac0 con 0x7f63e005c7c0 2021-03-09T09:25:56.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.540+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:25:56.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.540+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:25:56.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.541+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00554a0 con 0x7f63e005c7c0 2021-03-09T09:25:56.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:56.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:56.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:57.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:57.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:57.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:57.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:58.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:58.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:58.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:58.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:59.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:25:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:59.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:25:59.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:25:59.775+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:00.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:00.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:00.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:00.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:01.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:01.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:01.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6162 cs=0 l=1 rev1=1 rx=0x7f63f0080040 tx=0x7f63f0030ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:01.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6162 cs=0 l=1 rev1=1 rx=0x7f63f0080040 tx=0x7f63f0030ee0).stop 2021-03-09T09:26:01.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:01.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:01.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:01.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:01.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:26:01.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:01.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:01.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:01.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:01.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00281f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:01.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:01.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:01.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b820 con 0x7f63e00070a0 2021-03-09T09:26:01.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:01.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a750 con 0x7f6400145bd0 2021-03-09T09:26:01.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:01.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c2c0 con 0x7f63e0064730 2021-03-09T09:26:01.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.539+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:01.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.540+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00281f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:01.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.540+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:26:01.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.540+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:01.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.540+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:01.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.540+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:01.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.540+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:01.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.540+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:01.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:01.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:01.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:01.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:01.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:01.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:26:01.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:26:01.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:01.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:01.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0069660 con 0x7f63e0064730 2021-03-09T09:26:01.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:01.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:01.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63e0064730 2021-03-09T09:26:01.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:26:01.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:01.561 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:01.541+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16682 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:01.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.542+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00281f0 secure :-1 s=READY pgs=6174 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0071030).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:01.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.542+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:26:01.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.542+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.542+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0024210 con 0x7f63e0064730 2021-03-09T09:26:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.542+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.542+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.542+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:01.565 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:01.565 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:01.565 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:01.565 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:01.566 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:01.566 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:01.566 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:01.566 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:01.567 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:01.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0024210 con 0x7f63e0064730 2021-03-09T09:26:01.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.543+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:01.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.543+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:26:01.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001c7a0 con 0x7f63e0064730 2021-03-09T09:26:01.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:01.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:01.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:02.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:02.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:02.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:02.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:02.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:02.626+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:03.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:03.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:03.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:03.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:04.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:04.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:04.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:04.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:04.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:04.776+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:05.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:05.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:05.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:05.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:06.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00281f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:26:06.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00281f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:06.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00281f0 secure :-1 s=READY pgs=6174 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0071030).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:06.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00281f0 secure :-1 s=READY pgs=6174 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0071030).stop 2021-03-09T09:26:06.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:06.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:06.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00281f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:06.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00281f0 unknown :-1 s=CLOSED pgs=6174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:06.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:26:06.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:06.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:06.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:06.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:06.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:06.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:06.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:06.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de20 con 0x7f63e00070a0 2021-03-09T09:26:06.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:06.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f6400145bd0 2021-03-09T09:26:06.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:06.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00653b0 con 0x7f63e005c7c0 2021-03-09T09:26:06.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:06.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:06.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:06.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:06.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:06.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.542+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:06.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:26:06.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:06.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:06.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:06.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:06.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:06.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:06.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:06.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:06.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:06.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:06.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:26:06.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:26:06.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:06.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:06.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63e005c7c0 2021-03-09T09:26:06.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:06.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:06.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:26:06.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:26:06.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.543+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:06.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.544+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:26:06.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.544+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6186 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40099c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:06.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.544+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:06.557 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:06.544+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16721 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:06.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e005c7c0 2021-03-09T09:26:06.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.546+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:06.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.546+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:06.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.546+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:06.559 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:06.559 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:06.559 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:06.560 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:06.560 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:06.560 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:06.560 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:06.560 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:06.561 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:06.561 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:06.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b380 con 0x7f63e005c7c0 2021-03-09T09:26:06.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.546+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:06.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402d2a0 con 0x7f63e005c7c0 2021-03-09T09:26:06.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.547+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:26:06.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:06.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:06.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:07.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:07.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:07.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:07.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:08.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:08.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:08.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:08.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:09.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:09.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:09.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:09.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:09.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:09.776+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:10.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:10.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:10.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:10.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:11.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:11.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6186 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40099c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:11.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6186 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40099c0).stop 2021-03-09T09:26:11.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:11.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:11.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:11.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:11.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:26:11.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:11.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:11.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:11.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:11.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:11.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:11.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:11.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:26:11.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:11.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f6400145bd0 2021-03-09T09:26:11.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:11.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00131c0 con 0x7f63e0064730 2021-03-09T09:26:11.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.543+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:11.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:11.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:11.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:11.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:11.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:11.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:26:11.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:11.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:11.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:11.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.544+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:11.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:11.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:11.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:11.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:11.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:11.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:26:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:26:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:11.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f470 con 0x7f63e0064730 2021-03-09T09:26:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63e0064730 2021-03-09T09:26:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:26:11.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.545+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:11.559 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:11.546+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16759 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.546+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:26:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.546+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:11.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.546+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001e720 secure :-1 s=READY pgs=6202 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4022950).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:11.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401b2e0 con 0x7f63e0064730 2021-03-09T09:26:11.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.547+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:11.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.547+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:11.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.547+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:11.562 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:11.562 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:11.562 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:11.562 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:11.562 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:11.564 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f401b460 con 0x7f63e0064730 2021-03-09T09:26:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.547+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.547+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:26:11.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.548+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f401b5d0 con 0x7f63e0064730 2021-03-09T09:26:11.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:11.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:11.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:12.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:12.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:12.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:12.627+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:13.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:13.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:13.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:13.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:14.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:14.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:14.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:14.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:14.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:14.777+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:15.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:15.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:15.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:15.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:16.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:16.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:16.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001e720 secure :-1 s=READY pgs=6202 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4022950).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:16.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001e720 secure :-1 s=READY pgs=6202 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4022950).stop 2021-03-09T09:26:16.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:16.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:16.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e001e720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:16.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e001e720 unknown :-1 s=CLOSED pgs=6202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:16.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:26:16.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:16.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:16.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:16.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:16.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e00137c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:16.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:16.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:16.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066600 con 0x7f63e00070a0 2021-03-09T09:26:16.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:16.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065b90 con 0x7f6400145bd0 2021-03-09T09:26:16.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:16.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de20 con 0x7f63e005c7c0 2021-03-09T09:26:16.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:16.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e00137c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:16.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.546+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e00137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:16.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.547+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:16.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.547+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:16.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.547+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:16.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.547+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:26:16.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.547+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:16.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.547+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:16.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.547+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:16.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.547+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:16.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.547+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:16.558 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:16.548+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16798 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:16.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:16.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:16.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:16.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e00137c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:16.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e00137c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:16.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:26:16.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:26:16.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:16.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:16.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0077f50 con 0x7f63e005c7c0 2021-03-09T09:26:16.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:16.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:16.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:26:16.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:26:16.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:16.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:26:16.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:16.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.549+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6221 cs=0 l=1 rev1=1 rx=0x7f63e00191e0 tx=0x7f63fc0039e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:16.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.552+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e005c7c0 2021-03-09T09:26:16.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.552+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:16.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.552+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:16.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.552+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:16.564 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:16.564 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:16.564 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:16.565 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:16.565 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:16.565 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:16.565 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:16.566 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e005c7c0 2021-03-09T09:26:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.554+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.554+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:26:16.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e005c7c0 2021-03-09T09:26:16.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:16.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:16.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:17.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:17.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:17.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:17.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:18.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:18.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:18.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:18.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:19.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:19.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:19.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:19.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:19.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:19.778+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:20.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:20.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:20.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:20.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:26:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6221 cs=0 l=1 rev1=1 rx=0x7f63e00191e0 tx=0x7f63fc0039e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6221 cs=0 l=1 rev1=1 rx=0x7f63e00191e0 tx=0x7f63fc0039e0).stop 2021-03-09T09:26:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:21.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:21.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:26:21.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:21.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:21.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:21.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:21.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:21.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:21.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:21.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d700 con 0x7f63e00070a0 2021-03-09T09:26:21.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:21.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001dc80 con 0x7f6400145bd0 2021-03-09T09:26:21.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:21.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f63e0064730 2021-03-09T09:26:21.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:21.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:21.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:21.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:21.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:21.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:21.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:26:21.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:21.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.549+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:21.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.550+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:21.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.550+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:21.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.550+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:21.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:21.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:21.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:21.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:21.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:21.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:26:21.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:26:21.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:21.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:21.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00126c0 con 0x7f63e0064730 2021-03-09T09:26:21.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:21.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:21.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0077f50 con 0x7f63e0064730 2021-03-09T09:26:21.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:26:21.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:21.564 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:21.551+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16836 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.552+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6233 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc023e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.552+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:26:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.552+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.553+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065450 con 0x7f63e0064730 2021-03-09T09:26:21.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.553+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.553+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.553+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:21.568 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:21.569 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:21.569 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:21.569 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:21.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e0064730 2021-03-09T09:26:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.554+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.554+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:26:21.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02bbd0 con 0x7f63e0064730 2021-03-09T09:26:21.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:21.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:21.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:22.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:22.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:22.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:22.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:22.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:22.628+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:23.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:23.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:23.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:23.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:24.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:24.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:24.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:24.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:24.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:24.779+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:25.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:25.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:25.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:25.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:26.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:26:26.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:26.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6233 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc023e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:26.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6233 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc023e20).stop 2021-03-09T09:26:26.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:26.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:26.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:26.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:26.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:26:26.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:26.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:26.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:26.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:26.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:26.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:26.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:26.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019ca0 con 0x7f63e00070a0 2021-03-09T09:26:26.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:26.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f170 con 0x7f6400145bd0 2021-03-09T09:26:26.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:26.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029620 con 0x7f63e005c7c0 2021-03-09T09:26:26.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:26.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:26.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:26.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:26.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:26.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.552+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:26.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.553+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:26:26.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.553+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:26.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.553+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:26.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.553+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:26.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.553+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:26.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.553+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:26.563 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16873 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:26.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:26.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:26.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:26.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:26.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:26.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:26:26.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:26:26.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:26.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:26.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da30 con 0x7f63e005c7c0 2021-03-09T09:26:26.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:26.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:26.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00126c0 con 0x7f63e005c7c0 2021-03-09T09:26:26.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:26:26.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:26.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6246 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc0654c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:26.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:26:26.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.554+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:26.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03f280 con 0x7f63e005c7c0 2021-03-09T09:26:26.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.555+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:26.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.555+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:26.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.555+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:26.570 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:26.570 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:26.571 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:26.571 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:26.571 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:26.571 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:26.571 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:26.572 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:26.572 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:26.572 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:26.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e005c7c0 2021-03-09T09:26:26.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.557+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:26.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.557+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:26:26.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.558+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065b40 con 0x7f63e005c7c0 2021-03-09T09:26:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:26.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:27.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:27.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:27.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:27.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:28.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:28.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:28.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:28.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:29.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:29.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:29.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:29.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:29.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:29.780+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:30.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:30.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:30.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:30.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:31.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:31.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:31.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6246 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc0654c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:31.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6246 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc0654c0).stop 2021-03-09T09:26:31.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:31.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:31.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:31.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:31.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:26:31.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:31.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:31.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:31.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:31.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b870 con 0x7f63e00070a0 2021-03-09T09:26:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dc80 con 0x7f6400145bd0 2021-03-09T09:26:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000efa0 con 0x7f63e0064730 2021-03-09T09:26:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.554+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:31.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.555+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:31.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.555+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:31.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.555+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:31.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.555+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:31.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.556+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:31.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.556+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:26:31.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.556+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:31.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.556+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:31.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.556+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:31.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.556+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:31.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.556+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:31.566 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16911 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:31.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:31.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:31.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:31.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:31.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:31.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:26:31.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:26:31.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:31.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:31.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012300 con 0x7f63e0064730 2021-03-09T09:26:31.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:31.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:31.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f470 con 0x7f63e0064730 2021-03-09T09:26:31.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:26:31.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:31.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.557+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6262 cs=0 l=1 rev1=1 rx=0x7f63f4061ff0 tx=0x7f63f40035e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:31.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.558+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:26:31.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.558+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:31.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.558+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0064730 2021-03-09T09:26:31.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.558+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:31.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.558+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:31.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.558+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:31.573 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:31.573 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:31.573 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:31.574 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:31.574 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:31.574 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:31.575 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:31.575 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:31.575 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:31.575 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:31.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.559+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0064730 2021-03-09T09:26:31.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.559+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:31.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.559+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:26:31.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f63e0064730 2021-03-09T09:26:31.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:31.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:32.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:32.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:32.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:32.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:32.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:32.629+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:33.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:33.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:33.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:33.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:34.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:34.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:34.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:34.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:34.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:34.781+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:35.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:35.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:35.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:35.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:36.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:36.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:36.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6262 cs=0 l=1 rev1=1 rx=0x7f63f4061ff0 tx=0x7f63f40035e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:36.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6262 cs=0 l=1 rev1=1 rx=0x7f63f4061ff0 tx=0x7f63f40035e0).stop 2021-03-09T09:26:36.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:36.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:36.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:36.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:36.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:26:36.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:36.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:36.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:36.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:36.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:36.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:36.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:36.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:26:36.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:36.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f6400145bd0 2021-03-09T09:26:36.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:36.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b870 con 0x7f63e005c7c0 2021-03-09T09:26:36.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:36.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:36.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.557+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:36.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.558+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:36.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.558+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:36.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.558+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:36.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.558+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:26:36.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.558+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:36.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.558+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:36.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.558+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:36.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.558+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:36.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.558+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:36.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:36.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:36.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:36.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:36.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:36.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:26:36.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:26:36.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:36.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:36.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00126c0 con 0x7f63e005c7c0 2021-03-09T09:26:36.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:36.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:36.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da30 con 0x7f63e005c7c0 2021-03-09T09:26:36.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:26:36.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:36.571 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:36.559+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16952 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:36.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.560+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6271 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc03de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:36.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.560+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:26:36.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.560+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:36.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.561+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04bc70 con 0x7f63e005c7c0 2021-03-09T09:26:36.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.561+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:36.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.561+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:36.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.561+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:36.574 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:36.574 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:36.574 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:36.574 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:36.574 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:36.575 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:36.575 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:36.575 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:36.575 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:36.575 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:36.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04ba40 con 0x7f63e005c7c0 2021-03-09T09:26:36.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.562+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:36.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.562+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:26:36.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63e005c7c0 2021-03-09T09:26:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:36.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:37.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:37.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:37.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:37.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:38.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:38.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:38.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:38.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:39.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:39.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:39.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:39.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:39.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:39.782+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:40.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:40.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:40.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:40.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:41.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:41.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:41.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6271 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc03de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:41.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6271 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc03de00).stop 2021-03-09T09:26:41.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:41.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:41.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:41.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:41.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:26:41.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:41.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:41.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:41.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:41.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:41.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:41.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:41.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e00070a0 2021-03-09T09:26:41.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:41.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000efa0 con 0x7f6400145bd0 2021-03-09T09:26:41.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:41.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00260b0 con 0x7f63e0064730 2021-03-09T09:26:41.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:41.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:41.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:41.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.560+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:41.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.561+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:41.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.561+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:41.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.561+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:26:41.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.561+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:41.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.561+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:41.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.561+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:41.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.561+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:41.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.561+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:41.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:41.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:41.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:41.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:41.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:41.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:26:41.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:26:41.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:41.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:41.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T09:26:41.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:41.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:41.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00126c0 con 0x7f63e0064730 2021-03-09T09:26:41.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:26:41.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:41.575 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:41.562+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 16990 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:41.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.563+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ba20 secure :-1 s=READY pgs=6287 cs=0 l=1 rev1=1 rx=0x7f63fc0471e0 tx=0x7f63fc04f380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:41.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.563+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:26:41.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.563+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:41.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04ba40 con 0x7f63e0064730 2021-03-09T09:26:41.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.564+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:41.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.564+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:41.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.564+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:41.578 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:41.578 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:41.578 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:41.578 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:41.579 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:41.579 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:41.579 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:41.579 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:41.579 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:41.580 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:41.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.566+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04ba40 con 0x7f63e0064730 2021-03-09T09:26:41.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.566+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:41.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.566+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:26:41.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.566+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e0064730 2021-03-09T09:26:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:41.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:42.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:42.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:42.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:42.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:42.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:42.631+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:43.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:43.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:43.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:43.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:44.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:44.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:44.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:44.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:44.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:44.783+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:45.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:45.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:46.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:46.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:46.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ba20 secure :-1 s=READY pgs=6287 cs=0 l=1 rev1=1 rx=0x7f63fc0471e0 tx=0x7f63fc04f380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:46.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ba20 secure :-1 s=READY pgs=6287 cs=0 l=1 rev1=1 rx=0x7f63fc0471e0 tx=0x7f63fc04f380).stop 2021-03-09T09:26:46.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:46.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:46.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e000ba20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:46.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e000ba20 unknown :-1 s=CLOSED pgs=6287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:46.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:26:46.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:46.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:46.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:46.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:46.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:46.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:46.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:46.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f63e00070a0 2021-03-09T09:26:46.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:46.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ab10 con 0x7f6400145bd0 2021-03-09T09:26:46.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:46.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e020 con 0x7f63e005c7c0 2021-03-09T09:26:46.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.562+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:46.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:46.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:46.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:46.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:46.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:46.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:26:46.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:46.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:46.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:46.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:46.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.563+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:46.585 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:46.564+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17029 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:46.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.564+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:46.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:46.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:46.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:46.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:46.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:26:46.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:26:46.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:46.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:46.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025010 con 0x7f63e005c7c0 2021-03-09T09:26:46.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:46.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:46.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:26:46.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:26:46.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:46.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6299 cs=0 l=1 rev1=1 rx=0x7f63e0068ec0 tx=0x7f63fc04d5c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:46.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:26:46.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.565+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:46.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.567+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04ba40 con 0x7f63e005c7c0 2021-03-09T09:26:46.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.567+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:46.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.567+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:46.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.567+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:46.594 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:46.594 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:46.595 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:46.595 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:46.595 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:46.595 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:46.596 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:46.596 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:46.596 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:46.597 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:46.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04ba40 con 0x7f63e005c7c0 2021-03-09T09:26:46.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.569+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:46.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.569+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:26:46.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.570+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:26:46.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:46.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:46.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:47.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:47.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:47.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:47.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:48.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:48.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:48.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:48.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:49.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:49.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:49.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:49.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:49.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:49.784+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:50.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:50.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:50.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:50.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:51.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:26:51.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:51.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6299 cs=0 l=1 rev1=1 rx=0x7f63e0068ec0 tx=0x7f63fc04d5c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:51.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6299 cs=0 l=1 rev1=1 rx=0x7f63e0068ec0 tx=0x7f63fc04d5c0).stop 2021-03-09T09:26:51.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:51.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:51.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:51.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:51.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:26:51.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:51.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:51.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:51.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:51.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:51.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:51.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:51.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000eb10 con 0x7f63e00070a0 2021-03-09T09:26:51.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:51.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:26:51.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:51.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f63e0064730 2021-03-09T09:26:51.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:51.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:51.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:51.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:51.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:51.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:51.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:26:51.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:51.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.565+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:51.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.566+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:51.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.566+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:51.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.566+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:51.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:51.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:51.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:51.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:51.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:51.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:26:51.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:26:51.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:51.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:51.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b6c0 con 0x7f63e0064730 2021-03-09T09:26:51.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:51.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:51.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025010 con 0x7f63e0064730 2021-03-09T09:26:51.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:26:51.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:51.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:26:51.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.568+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:51.584 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17067 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:51.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.567+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6314 cs=0 l=1 rev1=1 rx=0x7f63fc02bff0 tx=0x7f63fc02dff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:51.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.570+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc021e70 con 0x7f63e0064730 2021-03-09T09:26:51.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.570+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:51.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.570+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:51.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.570+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:51.586 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:51.586 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:51.587 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:51.587 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:51.587 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:51.587 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:51.587 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:51.588 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:51.588 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:51.588 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:51.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc021e70 con 0x7f63e0064730 2021-03-09T09:26:51.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.574+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:51.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.574+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:26:51.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.574+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0064730 2021-03-09T09:26:51.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:51.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:51.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:52.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:52.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:52.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:52.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:52.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:52.631+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:53.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:53.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:53.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:53.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:54.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:54.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:54.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:54.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:54.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:54.785+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:26:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:55.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:55.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:55.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:56.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:26:56.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:26:56.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6314 cs=0 l=1 rev1=1 rx=0x7f63fc02bff0 tx=0x7f63fc02dff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:26:56.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6314 cs=0 l=1 rev1=1 rx=0x7f63fc02bff0 tx=0x7f63fc02dff0).stop 2021-03-09T09:26:56.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:56.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:26:56.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:26:56.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:26:56.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:26:56.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:56.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:26:56.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:56.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:26:56.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:26:56.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:26:56.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:56.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b870 con 0x7f63e00070a0 2021-03-09T09:26:56.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:56.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e960 con 0x7f6400145bd0 2021-03-09T09:26:56.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:26:56.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e005c7c0 2021-03-09T09:26:56.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:26:56.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:26:56.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.568+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.569+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:26:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.569+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:26:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.569+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:26:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.569+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:26:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.569+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:26:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.569+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:26:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.570+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:26:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.570+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:26:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.570+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:26:56.885 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:26:56.570+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17106 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:26:56.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:26:56.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:56.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:56.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:26:56.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:26:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:26:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:26:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:26:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:56.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d420 con 0x7f63e005c7c0 2021-03-09T09:26:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:26:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:26:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012300 con 0x7f63e005c7c0 2021-03-09T09:26:56.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:26:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6327 cs=0 l=1 rev1=1 rx=0x7f63f401c040 tx=0x7f63f4003bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:26:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:26:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.571+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:26:56.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.572+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40229c0 con 0x7f63e005c7c0 2021-03-09T09:26:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.572+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:26:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.572+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:26:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.572+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:26:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:26:56.891 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:26:56.892 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:56.892 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:26:56.892 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:26:56.892 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:26:56.892 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:26:56.893 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:26:56.893 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:26:56.893 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:26:56.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.574+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40229c0 con 0x7f63e005c7c0 2021-03-09T09:26:56.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.574+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:26:56.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.574+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:26:56.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.575+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f63e005c7c0 2021-03-09T09:26:56.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:56.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:56.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:57.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:57.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:57.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:57.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:58.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:58.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:59.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:59.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:26:59.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:59.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:26:59.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:26:59.786+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:00.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:00.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:00.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:00.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:01.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:27:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6327 cs=0 l=1 rev1=1 rx=0x7f63f401c040 tx=0x7f63f4003bb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6327 cs=0 l=1 rev1=1 rx=0x7f63f401c040 tx=0x7f63f4003bb0).stop 2021-03-09T09:27:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:01.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:01.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:01.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:01.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:27:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028bc0 con 0x7f63e00070a0 2021-03-09T09:27:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00176a0 con 0x7f6400145bd0 2021-03-09T09:27:01.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:01.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e020 con 0x7f63e0064730 2021-03-09T09:27:01.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:01.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:01.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:01.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:01.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:01.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:01.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:27:01.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.571+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.572+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.572+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.572+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:01.582 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17144 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:01.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:01.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:01.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:01.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:01.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:01.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:27:01.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:27:01.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:01.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:01.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00126c0 con 0x7f63e0064730 2021-03-09T09:27:01.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:01.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:01.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d420 con 0x7f63e0064730 2021-03-09T09:27:01.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:27:01.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.573+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:01.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.574+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6343 cs=0 l=1 rev1=1 rx=0x7f63f406b040 tx=0x7f63f4072f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:01.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.574+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:27:01.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.574+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:01.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.574+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402d4b0 con 0x7f63e0064730 2021-03-09T09:27:01.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.574+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:01.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.574+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:01.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.574+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:01.588 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:01.588 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:01.589 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:01.589 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:01.589 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:01.589 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:01.589 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:01.590 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:01.590 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:01.590 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:01.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4006a90 con 0x7f63e0064730 2021-03-09T09:27:01.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.576+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:01.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.576+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:27:01.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.577+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402d4b0 con 0x7f63e0064730 2021-03-09T09:27:01.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:01.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:01.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:02.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:02.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:02.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:02.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:02.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:02.633+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:03.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:03.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:03.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:03.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:04.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:04.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:04.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:04.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:04.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:04.787+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:05.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:05.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:05.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:05.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:06.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:06.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:06.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6343 cs=0 l=1 rev1=1 rx=0x7f63f406b040 tx=0x7f63f4072f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:06.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6343 cs=0 l=1 rev1=1 rx=0x7f63f406b040 tx=0x7f63f4072f70).stop 2021-03-09T09:27:06.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:27:06.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:06.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:06.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:06.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:06.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:06.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:06.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:06.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:27:06.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:06.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e710 con 0x7f6400145bd0 2021-03-09T09:27:06.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:06.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c4c0 con 0x7f63e005c7c0 2021-03-09T09:27:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.573+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.574+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:06.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.574+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:06.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.574+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:06.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.574+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:27:06.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.574+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:06.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.574+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:06.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:06.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:06.587 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17183 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:06.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:27:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:27:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:06.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:06.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d420 con 0x7f63e005c7c0 2021-03-09T09:27:06.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:06.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:06.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00126c0 con 0x7f63e005c7c0 2021-03-09T09:27:06.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:27:06.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.575+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:06.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.576+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:27:06.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.576+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:06.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.576+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6354 cs=0 l=1 rev1=1 rx=0x7f63f4017f80 tx=0x7f63f4046e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:06.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.577+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405c2f0 con 0x7f63e005c7c0 2021-03-09T09:27:06.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.577+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:06.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.577+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:06.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.577+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:06.593 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:06.593 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:06.593 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:06.594 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:06.594 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:06.594 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:06.594 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:06.594 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:06.595 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:06.595 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:06.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.577+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b520 con 0x7f63e005c7c0 2021-03-09T09:27:06.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.577+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:06.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.577+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:27:06.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.581+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022c30 con 0x7f63e005c7c0 2021-03-09T09:27:06.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:06.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:06.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:07.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:07.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:07.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:07.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:08.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:08.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:08.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:08.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:09.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:09.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:09.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:09.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:09.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:09.788+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:10.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:10.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:10.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:10.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:11.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:11.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:11.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6354 cs=0 l=1 rev1=1 rx=0x7f63f4017f80 tx=0x7f63f4046e10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:11.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6354 cs=0 l=1 rev1=1 rx=0x7f63f4017f80 tx=0x7f63f4046e10).stop 2021-03-09T09:27:11.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:11.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:11.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:11.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:11.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:27:11.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:11.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:11.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:11.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:11.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:11.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:11.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:11.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000efa0 con 0x7f63e00070a0 2021-03-09T09:27:11.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:11.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f6400145bd0 2021-03-09T09:27:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:27:11.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:11.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:11.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:11.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:11.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.576+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:11.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.577+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.577+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:27:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.577+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.577+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.577+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:11.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.577+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:11.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.577+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:11.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:11.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:11.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:11.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:27:11.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:27:11.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:11.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:11.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce90 con 0x7f63e0064730 2021-03-09T09:27:11.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:11.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:11.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b6c0 con 0x7f63e0064730 2021-03-09T09:27:11.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:27:11.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:11.604 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:11.578+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17221 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:11.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.579+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:27:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.579+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6369 cs=0 l=1 rev1=1 rx=0x7f63fc06dee0 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.579+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T09:27:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.579+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:11.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.579+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:11.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.579+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:11.606 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:11.606 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:11.607 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:11.607 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:11.607 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:11.607 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:11.607 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:11.608 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:11.608 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:11.608 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:11.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.580+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T09:27:11.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.580+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:11.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.580+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:27:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0064730 2021-03-09T09:27:11.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:11.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:11.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:12.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:12.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:12.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:12.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:12.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:12.635+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:13.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:13.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:13.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:13.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:14.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:14.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:14.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:14.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:14.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:14.789+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:15.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:15.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:15.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:15.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:16.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:16.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:16.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6369 cs=0 l=1 rev1=1 rx=0x7f63fc06dee0 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:16.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6369 cs=0 l=1 rev1=1 rx=0x7f63fc06dee0 tx=0x7f63fc003a50).stop 2021-03-09T09:27:16.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:16.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:16.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:16.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:16.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:27:16.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:16.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:16.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:16.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:16.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:16.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:16.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:16.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010130 con 0x7f63e00070a0 2021-03-09T09:27:16.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:16.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066a80 con 0x7f6400145bd0 2021-03-09T09:27:16.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:16.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f63e005c7c0 2021-03-09T09:27:16.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:16.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:16.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:16.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:16.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:16.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:16.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:27:16.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:16.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.579+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:16.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:16.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:16.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:16.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:16.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:16.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:16.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:16.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:27:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:27:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f0b0 con 0x7f63e005c7c0 2021-03-09T09:27:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d420 con 0x7f63e005c7c0 2021-03-09T09:27:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:27:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.580+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.581+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:27:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.581+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6380 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4072cd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:16.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.581+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:16.596 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:16.581+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17260 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:16.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048640 con 0x7f63e005c7c0 2021-03-09T09:27:16.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.582+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:16.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.582+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.582+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:16.600 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:16.600 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:16.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40098c0 con 0x7f63e005c7c0 2021-03-09T09:27:16.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.582+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:16.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.582+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:27:16.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.584+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4053d90 con 0x7f63e005c7c0 2021-03-09T09:27:16.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:16.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:16.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:17.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:17.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:17.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:17.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:18.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:18.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:18.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:18.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:19.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:19.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:19.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:19.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:19.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:19.790+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:20.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:20.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:20.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:20.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:21.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:21.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:21.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6380 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4072cd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:21.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6380 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4072cd0).stop 2021-03-09T09:27:21.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:21.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:21.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:21.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:21.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:27:21.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:21.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:21.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:21.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:21.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:21.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:21.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:21.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e00070a0 2021-03-09T09:27:21.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:21.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400145bd0 2021-03-09T09:27:21.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:21.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000efa0 con 0x7f63e0064730 2021-03-09T09:27:21.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:21.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:21.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:21.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:21.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.582+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:21.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.583+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:21.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.583+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:27:21.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.583+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:21.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.583+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:21.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.583+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:21.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.583+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:21.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.583+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:21.593 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17298 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:21.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:21.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:21.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:21.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:21.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:21.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:27:21.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:27:21.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:21.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:21.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022240 con 0x7f63e0064730 2021-03-09T09:27:21.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:21.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:21.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f0b0 con 0x7f63e0064730 2021-03-09T09:27:21.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:27:21.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.584+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:21.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.585+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6397 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404d300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:21.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.585+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:27:21.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.585+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:21.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.586+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048200 con 0x7f63e0064730 2021-03-09T09:27:21.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.586+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.586+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.586+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:21.600 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:21.600 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:21.600 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:21.600 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:21.601 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:21.601 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:21.601 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:21.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4048200 con 0x7f63e0064730 2021-03-09T09:27:21.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.587+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:21.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.587+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:27:21.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63e0064730 2021-03-09T09:27:21.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:21.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:21.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:22.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:22.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:22.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:22.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:22.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:22.636+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:23.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:23.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:23.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:23.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:24.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:24.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:24.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:24.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:24.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:24.791+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:25.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:25.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:25.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:25.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:26.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:27:26.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:26.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6397 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404d300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:26.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6397 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404d300).stop 2021-03-09T09:27:26.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:26.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:26.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:26.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:26.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:27:26.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:26.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:26.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:26.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:26.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:26.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:26.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:26.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f3a0 con 0x7f63e00070a0 2021-03-09T09:27:26.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:26.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024400 con 0x7f6400145bd0 2021-03-09T09:27:26.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:26.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:27:26.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:26.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:26.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:26.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:26.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:26.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:26.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:27:26.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:26.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.585+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:26.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.586+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:26.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.586+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:26.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.586+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:26.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.586+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:26.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:26.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:26.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:26.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:26.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:27:26.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:27:26.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:26.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:26.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d420 con 0x7f63e005c7c0 2021-03-09T09:27:26.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:26.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:26.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022240 con 0x7f63e005c7c0 2021-03-09T09:27:26.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:27:26.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:26.601 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17337 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:26.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6411 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f402c580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:27:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.587+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404be10 con 0x7f63e005c7c0 2021-03-09T09:27:26.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.588+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:26.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.588+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:26.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.588+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:26.603 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:26.603 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:26.604 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:26.604 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:26.604 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:26.604 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:26.605 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:26.605 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:26.605 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:26.605 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:26.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.589+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404be10 con 0x7f63e005c7c0 2021-03-09T09:27:26.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.589+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:26.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.589+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:27:26.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.590+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402d2a0 con 0x7f63e005c7c0 2021-03-09T09:27:26.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:26.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:26.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:27.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:27.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:27.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:27.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:28.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:28.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:28.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:28.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:29.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:29.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:29.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:29.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:29.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:29.792+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:30.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:30.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:30.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:30.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:31.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:27:31.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:31.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6411 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f402c580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:31.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6411 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f402c580).stop 2021-03-09T09:27:31.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:31.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:31.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:31.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:31.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:27:31.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:31.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:31.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:31.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:31.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:31.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:31.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:31.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f63e00070a0 2021-03-09T09:27:31.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:31.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f100 con 0x7f6400145bd0 2021-03-09T09:27:31.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:31.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e450 con 0x7f63e0064730 2021-03-09T09:27:31.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.587+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:31.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.588+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:31.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.588+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:31.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.588+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:31.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.588+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:31.597 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:31.589+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17375 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:31.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.590+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:31.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.590+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:27:31.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.590+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:31.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.590+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:31.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.590+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:31.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.590+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:31.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.590+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:31.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:31.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:31.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:31.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:31.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:31.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:27:31.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:27:31.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:31.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:31.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00126c0 con 0x7f63e0064730 2021-03-09T09:27:31.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:31.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:31.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce90 con 0x7f63e0064730 2021-03-09T09:27:31.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:27:31.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:31.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6430 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc04d580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:31.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:27:31.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.591+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:31.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.594+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc006550 con 0x7f63e0064730 2021-03-09T09:27:31.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.594+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:31.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.594+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:31.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.594+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:31.605 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:31.605 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:31.606 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:31.606 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:31.606 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:31.606 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:31.606 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:31.607 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:31.607 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:31.607 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:31.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.595+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc006550 con 0x7f63e0064730 2021-03-09T09:27:31.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.595+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:31.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.595+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:27:31.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.597+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0064730 2021-03-09T09:27:31.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:31.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:31.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:32.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:32.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:32.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:32.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:32.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:32.637+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:33.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:33.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:33.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:33.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:34.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:34.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:34.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:34.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:34.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:34.793+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:35.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:35.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:35.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:35.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:36.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:36.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:36.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6430 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc04d580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:36.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6430 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc04d580).stop 2021-03-09T09:27:36.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:36.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:36.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:36.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:36.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:27:36.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:36.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:36.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:36.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:36.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:36.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:36.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:36.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e00070a0 2021-03-09T09:27:36.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:36.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066a80 con 0x7f6400145bd0 2021-03-09T09:27:36.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:36.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000efa0 con 0x7f63e005c7c0 2021-03-09T09:27:36.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:36.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:36.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:36.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:36.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.590+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:36.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.591+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:36.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.591+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:27:36.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.591+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:36.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.591+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:36.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:36.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:36.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:36.602 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17415 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:36.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:36.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:36.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:36.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:36.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:36.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:27:36.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:27:36.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:36.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:36.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c920 con 0x7f63e005c7c0 2021-03-09T09:27:36.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:36.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:36.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00126c0 con 0x7f63e005c7c0 2021-03-09T09:27:36.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:27:36.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.592+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:36.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.593+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:27:36.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.593+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6440 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc02c400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:36.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.593+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:36.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.594+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065450 con 0x7f63e005c7c0 2021-03-09T09:27:36.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.594+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:36.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.594+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:36.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.594+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:36.608 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:36.608 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:36.609 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:36.609 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:36.609 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:36.609 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:36.609 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:36.610 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:36.610 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:36.610 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:36.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.595+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02c860 con 0x7f63e005c7c0 2021-03-09T09:27:36.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.595+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:36.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.595+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:27:36.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065b40 con 0x7f63e005c7c0 2021-03-09T09:27:36.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:36.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:36.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:37.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:37.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:37.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:38.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:38.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:38.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:38.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:39.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:39.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:39.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:39.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:39.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:39.794+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:40.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:40.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:40.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:40.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:41.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:41.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:41.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.592+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6440 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc02c400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:41.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6440 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc02c400).stop 2021-03-09T09:27:41.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:41.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:41.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:41.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:41.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:27:41.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:41.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:41.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:41.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:41.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00017f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:41.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:41.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:41.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:27:41.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:41.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f400 con 0x7f6400145bd0 2021-03-09T09:27:41.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:41.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a0c0 con 0x7f63e0064730 2021-03-09T09:27:41.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:41.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:41.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:41.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:41.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.593+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:41.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:41.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:27:41.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:41.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:41.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:41.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:41.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:41.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:41.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:41.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:41.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:41.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:41.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:27:41.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:27:41.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:41.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:41.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T09:27:41.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:41.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:41.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0069660 con 0x7f63e0064730 2021-03-09T09:27:41.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:27:41.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.594+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:41.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.595+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00017f0 secure :-1 s=READY pgs=6448 cs=0 l=1 rev1=1 rx=0x7f63f0080040 tx=0x7f63f006f010).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:41.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.595+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:27:41.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.595+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:41.610 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:41.595+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17452 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:41.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0064730 2021-03-09T09:27:41.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.596+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:41.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.596+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:41.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.596+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:41.612 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:41.612 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:41.612 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:41.612 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:41.612 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:41.613 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:41.613 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:41.613 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:41.613 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:41.614 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:41.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T09:27:41.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.596+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:41.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.596+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:27:41.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.598+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T09:27:41.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:41.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:41.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:42.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:42.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:42.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:42.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:42.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:42.638+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:43.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:43.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:43.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:43.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:44.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:44.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:44.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:44.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:44.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:44.795+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:45.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:45.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:45.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:45.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:46.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:46.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:46.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00017f0 secure :-1 s=READY pgs=6448 cs=0 l=1 rev1=1 rx=0x7f63f0080040 tx=0x7f63f006f010).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:46.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00017f0 secure :-1 s=READY pgs=6448 cs=0 l=1 rev1=1 rx=0x7f63f0080040 tx=0x7f63f006f010).stop 2021-03-09T09:27:46.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:46.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:46.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00017f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:46.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00017f0 unknown :-1 s=CLOSED pgs=6448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:46.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:27:46.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:46.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:46.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:46.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:46.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:46.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:46.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:46.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e450 con 0x7f63e00070a0 2021-03-09T09:27:46.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:46.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ab10 con 0x7f6400145bd0 2021-03-09T09:27:46.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:46.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000efa0 con 0x7f63e005c7c0 2021-03-09T09:27:46.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:46.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:46.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:46.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:46.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:46.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:46.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:27:46.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:46.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.596+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:46.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:46.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:46.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:46.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:46.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:46.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:46.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:46.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:46.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:27:46.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:27:46.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:46.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:46.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00777c0 con 0x7f63e005c7c0 2021-03-09T09:27:46.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:46.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:46.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c920 con 0x7f63e005c7c0 2021-03-09T09:27:46.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:27:46.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.597+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:46.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.598+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:27:46.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.598+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:46.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.598+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6461 cs=0 l=1 rev1=1 rx=0x7f63fc02c5c0 tx=0x7f63fc012f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:46.611 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:46.598+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17492 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:46.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e005c7c0 2021-03-09T09:27:46.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.599+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:46.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.599+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:46.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.599+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:46.612 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:46.613 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:46.613 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:46.613 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:46.613 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:46.613 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:46.614 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:46.614 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:46.614 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:46.614 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:46.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032500 con 0x7f63e005c7c0 2021-03-09T09:27:46.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.599+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:46.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.599+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:27:46.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.600+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63e005c7c0 2021-03-09T09:27:46.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:46.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:46.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:47.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:47.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:47.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:47.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:48.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:48.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:48.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:48.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:49.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:49.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:49.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:49.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:49.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:49.796+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:50.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:50.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:50.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:50.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:51.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:27:51.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:51.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6461 cs=0 l=1 rev1=1 rx=0x7f63fc02c5c0 tx=0x7f63fc012f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:51.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6461 cs=0 l=1 rev1=1 rx=0x7f63fc02c5c0 tx=0x7f63fc012f50).stop 2021-03-09T09:27:51.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:51.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:51.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:51.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:51.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:27:51.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:51.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:51.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:51.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:51.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:51.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:51.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:51.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cf0 con 0x7f63e00070a0 2021-03-09T09:27:51.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:51.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400145bd0 2021-03-09T09:27:51.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:51.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067950 con 0x7f63e0064730 2021-03-09T09:27:51.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:51.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:51.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.599+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:51.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:51.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:27:51.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:51.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:51.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:51.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:51.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:51.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:51.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:51.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:51.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:51.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:51.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:27:51.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:27:51.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:51.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:51.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00126c0 con 0x7f63e0064730 2021-03-09T09:27:51.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:51.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:51.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0064730 2021-03-09T09:27:51.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.601+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:27:51.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.601+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:51.619 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:51.601+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17529 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:51.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.602+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:27:51.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.602+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6474 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f00728c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:51.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.602+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:51.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.602+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0072af0 con 0x7f63e0064730 2021-03-09T09:27:51.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.602+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:51.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.602+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:51.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.602+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:51.622 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:51.623 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:51.623 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:51.623 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:51.624 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:51.624 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:51.624 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:51.625 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:51.625 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:51.626 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:51.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.603+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0072af0 con 0x7f63e0064730 2021-03-09T09:27:51.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.603+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:51.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.603+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:27:51.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0072d20 con 0x7f63e0064730 2021-03-09T09:27:51.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:51.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:51.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:52.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:52.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:52.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:52.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:52.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:52.640+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:53.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:53.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:53.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:53.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:54.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:54.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:54.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:54.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:54.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:54.797+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:27:55.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:55.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:55.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:55.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:27:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:27:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6474 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f00728c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:27:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6474 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f00728c0).stop 2021-03-09T09:27:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:27:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:27:56.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:27:56.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:27:56.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:56.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:27:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:27:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:27:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:27:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e00070a0 2021-03-09T09:27:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c4c0 con 0x7f6400145bd0 2021-03-09T09:27:56.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:27:56.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e450 con 0x7f63e005c7c0 2021-03-09T09:27:56.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:27:56.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:27:56.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:56.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:27:56.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:27:56.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:27:56.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:27:56.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:27:56.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:27:56.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:27:56.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:27:56.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.602+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:27:56.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:27:56.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:56.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:56.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:27:56.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:27:56.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:27:56.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:27:56.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:27:56.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:56.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006a070 con 0x7f63e005c7c0 2021-03-09T09:27:56.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:27:56.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:27:56.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00126c0 con 0x7f63e005c7c0 2021-03-09T09:27:56.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:27:56.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:56.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6489 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f0038e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:27:56.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:27:56.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.603+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:27:56.625 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:27:56.604+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17569 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:27:56.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0072480 con 0x7f63e005c7c0 2021-03-09T09:27:56.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.604+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:27:56.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.604+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:27:56.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.604+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:27:56.628 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:27:56.629 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:27:56.629 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:56.630 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:27:56.630 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:27:56.630 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:27:56.631 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:27:56.631 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:27:56.632 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:27:56.632 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:27:56.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00725e0 con 0x7f63e005c7c0 2021-03-09T09:27:56.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.604+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:27:56.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.604+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:27:56.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.606+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63e005c7c0 2021-03-09T09:27:56.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:56.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:56.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:57.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:57.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:57.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:57.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:58.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:58.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:58.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:58.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:59.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:59.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:27:59.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:59.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:27:59.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:27:59.798+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:00.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:00.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:00.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:00.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:01.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:01.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:01.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6489 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f0038e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:01.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6489 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f0038e80).stop 2021-03-09T09:28:01.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:01.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:01.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:01.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:01.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:28:01.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.604+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:01.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:01.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:01.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:01.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:01.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:01.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:01.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:28:01.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:01.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d770 con 0x7f6400145bd0 2021-03-09T09:28:01.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:01.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e0064730 2021-03-09T09:28:01.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:01.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:01.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:01.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:01.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.605+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:01.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:01.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:28:01.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:01.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:01.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:01.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:01.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:01.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:01.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:01.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:01.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:28:01.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:28:01.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:01.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:01.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a330 con 0x7f63e0064730 2021-03-09T09:28:01.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:01.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:01.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.606+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006a070 con 0x7f63e0064730 2021-03-09T09:28:01.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:28:01.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:01.621 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17606 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:01.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:28:01.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6502 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0072f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:01.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:01.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005e470 con 0x7f63e0064730 2021-03-09T09:28:01.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:01.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:01.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:01.623 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:01.624 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:01.624 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:01.624 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:01.624 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:01.625 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:01.625 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:01.625 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:01.625 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:01.625 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:01.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005e290 con 0x7f63e0064730 2021-03-09T09:28:01.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:01.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.607+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:28:01.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.609+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0066a70 con 0x7f63e0064730 2021-03-09T09:28:01.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:01.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:01.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:02.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:02.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:02.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:02.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:02.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:02.641+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:03.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:03.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:03.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:03.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:04.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:04.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:04.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:04.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:04.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:04.799+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:05.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:05.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:05.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:05.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:06.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:06.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:06.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6502 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0072f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:06.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6502 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0072f80).stop 2021-03-09T09:28:06.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:06.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:06.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:06.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:06.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:28:06.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:06.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:06.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:06.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:06.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:06.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:06.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:06.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e00070a0 2021-03-09T09:28:06.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:06.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ab10 con 0x7f6400145bd0 2021-03-09T09:28:06.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:06.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e270 con 0x7f63e005c7c0 2021-03-09T09:28:06.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:06.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:06.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:06.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:06.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:06.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:06.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:28:06.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:06.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.608+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:06.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:06.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:06.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:06.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:06.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:06.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:06.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:06.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:06.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:28:06.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:28:06.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:06.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:06.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019110 con 0x7f63e005c7c0 2021-03-09T09:28:06.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:06.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:06.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a330 con 0x7f63e005c7c0 2021-03-09T09:28:06.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:28:06.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.609+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:06.624 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:06.610+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17646 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:06.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.610+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6520 cs=0 l=1 rev1=1 rx=0x7f63e0025140 tx=0x7f63f007ff30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:06.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.610+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:28:06.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.610+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:06.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f63e005c7c0 2021-03-09T09:28:06.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.616+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:06.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.616+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:06.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.616+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:06.627 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:06.627 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:06.628 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:06.628 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:06.628 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:06.628 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:06.628 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:06.629 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:06.629 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:06.629 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:06.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005e6a0 con 0x7f63e005c7c0 2021-03-09T09:28:06.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.616+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:06.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.616+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:28:06.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0072a80 con 0x7f63e005c7c0 2021-03-09T09:28:06.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:06.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:06.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:07.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:07.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:08.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:08.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:08.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:08.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:09.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:09.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:09.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:09.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:09.800+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:10.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:10.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:10.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:10.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6520 cs=0 l=1 rev1=1 rx=0x7f63e0025140 tx=0x7f63f007ff30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:11.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6520 cs=0 l=1 rev1=1 rx=0x7f63e0025140 tx=0x7f63f007ff30).stop 2021-03-09T09:28:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:11.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:28:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:11.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0030060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068ae0 con 0x7f63e00070a0 2021-03-09T09:28:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:11.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e450 con 0x7f6400145bd0 2021-03-09T09:28:11.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:11.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0064730 2021-03-09T09:28:11.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:11.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:11.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.610+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:11.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.611+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:11.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.611+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.612+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0030060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.612+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:28:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.612+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:11.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.612+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:11.621 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:11.612+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17683 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:11.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.613+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:11.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.613+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:11.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.613+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:11.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:11.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:11.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:11.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:11.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:11.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:28:11.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:28:11.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:11.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:11.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006a070 con 0x7f63e0064730 2021-03-09T09:28:11.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:11.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:11.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019110 con 0x7f63e0064730 2021-03-09T09:28:11.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:28:11.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:11.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0030060 secure :-1 s=READY pgs=6540 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:11.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:28:11.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:11.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007bed0 con 0x7f63e0064730 2021-03-09T09:28:11.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:11.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:11.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.614+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:11.628 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:11.629 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:11.629 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:11.629 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:11.629 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:11.629 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:11.630 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:11.630 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:11.630 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:11.630 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:11.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.617+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0064730 2021-03-09T09:28:11.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.617+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:11.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.617+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f006da50 con 0x7f63e0064730 2021-03-09T09:28:11.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.618+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:28:11.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:11.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:11.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:12.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:12.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:12.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:12.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:12.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:12.643+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:13.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:13.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:13.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:13.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:14.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:14.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:14.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:14.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:14.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:14.801+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:15.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:15.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:15.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:15.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:16.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0030060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:16.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0030060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:16.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0030060 secure :-1 s=READY pgs=6540 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:16.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0030060 secure :-1 s=READY pgs=6540 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f000df50).stop 2021-03-09T09:28:16.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:16.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:16.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e0030060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:16.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e0030060 unknown :-1 s=CLOSED pgs=6540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:16.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:28:16.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:16.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:16.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:16.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:16.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:16.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:16.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:16.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e00070a0 2021-03-09T09:28:16.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:16.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065a30 con 0x7f6400145bd0 2021-03-09T09:28:16.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:16.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ab10 con 0x7f63e005c7c0 2021-03-09T09:28:16.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:16.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:16.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:16.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:16.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.613+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:16.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.614+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:16.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.614+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:28:16.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.614+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:16.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.614+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:16.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.614+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:16.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.614+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:16.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.614+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:16.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:16.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:16.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:16.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:16.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:16.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:28:16.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:28:16.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:16.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:16.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068890 con 0x7f63e005c7c0 2021-03-09T09:28:16.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:16.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:16.628 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17723 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:16.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00777c0 con 0x7f63e005c7c0 2021-03-09T09:28:16.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:28:16.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.615+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:16.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.616+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:28:16.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.616+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:16.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.616+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6548 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc02c220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:16.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046b60 con 0x7f63e005c7c0 2021-03-09T09:28:16.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.616+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:16.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.616+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:16.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.616+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:16.632 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:16.632 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:16.632 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:16.632 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:16.632 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:16.633 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:16.633 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:16.633 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:16.633 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:16.634 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:16.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046b60 con 0x7f63e005c7c0 2021-03-09T09:28:16.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.618+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:16.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.618+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:28:16.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:28:16.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:16.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:16.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:17.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:17.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:17.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:17.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:18.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:18.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:18.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:18.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:19.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:19.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:19.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:19.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:19.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:19.802+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:20.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:20.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:20.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:20.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:21.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:28:21.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:21.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6548 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc02c220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:21.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6548 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc02c220).stop 2021-03-09T09:28:21.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:21.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:21.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:21.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:21.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:28:21.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:21.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:21.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:21.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:21.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:21.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:21.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:21.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e450 con 0x7f63e00070a0 2021-03-09T09:28:21.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:21.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024fd0 con 0x7f6400145bd0 2021-03-09T09:28:21.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:21.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e0064730 2021-03-09T09:28:21.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:21.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:21.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:21.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:21.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:28:21.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:21.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:21.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:21.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:21.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:21.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:21.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.616+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:21.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:21.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:21.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:21.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:21.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:21.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:28:21.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:28:21.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:21.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:21.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00241d0 con 0x7f63e0064730 2021-03-09T09:28:21.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:21.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:21.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006a070 con 0x7f63e0064730 2021-03-09T09:28:21.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:28:21.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.617+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:21.635 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:21.618+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17750 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:21.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.618+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6560 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f0042f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:21.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.618+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:28:21.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.618+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:21.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e430 con 0x7f63e0064730 2021-03-09T09:28:21.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.619+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:21.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.619+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:21.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.619+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:21.637 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:21.638 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:21.638 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:21.638 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:21.638 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:21.638 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:21.639 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:21.639 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:21.639 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:21.639 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:21.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.621+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63e0064730 2021-03-09T09:28:21.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.621+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:21.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.621+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:28:21.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.622+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042320 con 0x7f63e0064730 2021-03-09T09:28:21.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:21.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:21.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:22.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:22.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:22.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:22.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:22.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:22.644+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:23.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:23.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:23.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:23.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:24.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:24.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:24.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:24.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:24.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:24.803+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:25.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:25.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:25.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:25.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:26.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:26.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:26.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6560 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f0042f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6560 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f0042f00).stop 2021-03-09T09:28:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:28:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:26.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:26.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:26.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:26.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:26.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f9f0 con 0x7f63e00070a0 2021-03-09T09:28:26.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:26.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400145bd0 2021-03-09T09:28:26.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:26.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:28:26.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:26.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:26.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:26.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:26.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.619+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:26.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.620+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:26.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.620+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:28:26.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.620+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:26.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.620+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:26.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.620+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.620+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.620+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:26.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:28:26.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:28:26.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:26.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:26.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a710 con 0x7f63e005c7c0 2021-03-09T09:28:26.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:26.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:26.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d420 con 0x7f63e005c7c0 2021-03-09T09:28:26.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:28:26.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:26.634 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17800 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:26.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6574 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:26.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:28:26.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.621+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:26.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.622+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009980 con 0x7f63e005c7c0 2021-03-09T09:28:26.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.622+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:26.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.622+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:26.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.622+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:26.637 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:26.637 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:26.637 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:26.637 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:26.638 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:26.638 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:26.638 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:26.638 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:26.638 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:26.639 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:26.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.623+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63e005c7c0 2021-03-09T09:28:26.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.623+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:26.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.623+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:28:26.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.624+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e005c7c0 2021-03-09T09:28:26.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:26.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:26.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:27.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:27.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:27.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:27.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:28.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:28.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:28.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:28.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:29.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:29.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:29.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:29.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:29.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:29.804+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:30.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:30.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:30.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:30.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:31.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:31.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:31.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6574 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:31.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6574 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4017ed0).stop 2021-03-09T09:28:31.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:31.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:31.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:31.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:31.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:28:31.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:31.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:31.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:31.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:31.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:31.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:31.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:31.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e450 con 0x7f63e00070a0 2021-03-09T09:28:31.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:31.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400145bd0 2021-03-09T09:28:31.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:31.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63e0064730 2021-03-09T09:28:31.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.622+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:31.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:31.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:31.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:28:31.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.623+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.624+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:31.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:31.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:31.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:31.633 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17837 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:31.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:31.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:31.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:28:31.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:28:31.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:31.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:31.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001de00 con 0x7f63e0064730 2021-03-09T09:28:31.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:31.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:31.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068890 con 0x7f63e0064730 2021-03-09T09:28:31.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:28:31.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.625+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:31.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.626+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6590 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63fc067010).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:31.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.626+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:28:31.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.626+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:31.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00f140 con 0x7f63e0064730 2021-03-09T09:28:31.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.627+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:31.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.627+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:31.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.627+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:31.639 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:31.639 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:31.640 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:31.640 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:31.640 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:31.640 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:31.640 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:31.641 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:31.641 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:31.641 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:31.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00f140 con 0x7f63e0064730 2021-03-09T09:28:31.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.627+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:31.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.627+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:28:31.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e0064730 2021-03-09T09:28:31.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:31.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:31.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:32.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:32.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:32.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:32.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:32.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:32.645+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:33.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:33.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:33.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:33.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:34.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:34.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:34.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:34.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:34.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:34.805+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:35.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:35.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:35.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:35.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:36.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6590 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63fc067010).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6590 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63fc067010).stop 2021-03-09T09:28:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:28:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e890 con 0x7f63e00070a0 2021-03-09T09:28:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00658d0 con 0x7f6400145bd0 2021-03-09T09:28:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e005c7c0 2021-03-09T09:28:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:36.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:36.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:28:36.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:36.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.625+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:36.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:36.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:36.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:36.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:36.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:36.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:36.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:28:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:28:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019110 con 0x7f63e005c7c0 2021-03-09T09:28:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:36.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:36.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a710 con 0x7f63e005c7c0 2021-03-09T09:28:36.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:28:36.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.626+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:36.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.627+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6599 cs=0 l=1 rev1=1 rx=0x7f63f402f000 tx=0x7f63f4005210).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:36.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.627+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:28:36.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.627+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:36.640 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:36.627+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17876 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:36.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e005c7c0 2021-03-09T09:28:36.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.628+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:36.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.628+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:36.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.628+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:36.641 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:36.642 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:36.642 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:36.642 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:36.642 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:36.648 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:36.648 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:36.648 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:36.648 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:36.649 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005b60 con 0x7f63e005c7c0 2021-03-09T09:28:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.628+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.628+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:28:36.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022870 con 0x7f63e005c7c0 2021-03-09T09:28:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:36.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:36.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:37.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:37.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:37.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:37.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:38.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:38.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:38.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:39.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:39.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:39.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:39.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:39.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:39.806+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:40.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:40.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:40.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:41.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:41.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:41.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6599 cs=0 l=1 rev1=1 rx=0x7f63f402f000 tx=0x7f63f4005210).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:41.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6599 cs=0 l=1 rev1=1 rx=0x7f63f402f000 tx=0x7f63f4005210).stop 2021-03-09T09:28:41.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:41.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:28:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:41.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:41.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f9f0 con 0x7f63e00070a0 2021-03-09T09:28:41.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:41.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400145bd0 2021-03-09T09:28:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e450 con 0x7f63e0064730 2021-03-09T09:28:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.628+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.629+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.629+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:28:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.629+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:41.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.629+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:41.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.630+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:41.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.630+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:41.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.630+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:41.639 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:41.630+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17914 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:41.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:41.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:41.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:41.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:41.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:41.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:28:41.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:28:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.631+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006aab0 con 0x7f63e0064730 2021-03-09T09:28:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.632+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.632+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.632+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001de00 con 0x7f63e0064730 2021-03-09T09:28:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.632+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:28:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.632+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.632+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6623 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc013290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:41.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.632+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:28:41.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.632+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:41.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.633+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T09:28:41.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.633+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:41.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.633+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:41.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.633+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:41.646 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:41.646 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:41.646 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:41.646 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:41.648 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:41.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T09:28:41.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.635+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:41.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.635+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:28:41.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.637+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e0064730 2021-03-09T09:28:41.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:41.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:41.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:42.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:42.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:42.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:42.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:42.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:42.647+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:43.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:43.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:43.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:44.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:44.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:44.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:44.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:44.807+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:45.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:45.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:45.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:45.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:46.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:46.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:46.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6623 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc013290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:46.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6623 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc013290).stop 2021-03-09T09:28:46.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:46.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:46.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:46.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:46.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:28:46.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:46.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:46.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:46.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:46.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:46.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:46.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:46.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008850 con 0x7f63e00070a0 2021-03-09T09:28:46.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:46.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e890 con 0x7f6400145bd0 2021-03-09T09:28:46.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:46.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ed70 con 0x7f63e005c7c0 2021-03-09T09:28:46.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:46.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:46.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.632+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:46.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.633+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.633+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.633+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.633+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:28:46.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.633+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:46.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.633+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:46.649 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:46.633+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17953 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:46.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.634+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.634+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.634+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:28:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:28:46.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001de00 con 0x7f63e005c7c0 2021-03-09T09:28:46.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019110 con 0x7f63e005c7c0 2021-03-09T09:28:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:28:46.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:46.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:28:46.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.635+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6638 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40036c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:46.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.636+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:46.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.641+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034da0 con 0x7f63e005c7c0 2021-03-09T09:28:46.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.641+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:46.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.641+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:46.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.641+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:46.663 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:46.664 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:46.664 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:46.665 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:46.665 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:46.665 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:46.666 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:46.666 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:46.667 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:46.667 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:46.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.641+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034960 con 0x7f63e005c7c0 2021-03-09T09:28:46.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.641+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:46.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.641+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:28:46.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.643+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4034960 con 0x7f63e005c7c0 2021-03-09T09:28:46.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:46.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:47.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:47.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:47.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:47.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:48.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:48.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:48.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:48.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:49.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:49.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:49.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:49.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:49.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:49.808+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:50.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:50.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:50.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:50.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:51.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:28:51.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:51.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6638 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40036c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6638 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40036c0).stop 2021-03-09T09:28:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:51.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:51.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:51.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:28:51.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:51.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:51.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:51.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:51.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:51.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:51.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:51.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:28:51.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:51.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400145bd0 2021-03-09T09:28:51.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:51.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00140b0 con 0x7f63e0064730 2021-03-09T09:28:51.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:51.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:51.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.635+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:51.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.636+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:51.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.636+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:28:51.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.636+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:51.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.636+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:51.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.636+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:51.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.636+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:51.645 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:51.636+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 17991 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:51.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:51.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:51.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:51.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:28:51.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:28:51.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:51.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:51.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0030400 con 0x7f63e0064730 2021-03-09T09:28:51.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:51.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:51.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00241d0 con 0x7f63e0064730 2021-03-09T09:28:51.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:28:51.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.637+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:51.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.638+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6645 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f0013b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:51.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.638+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:28:51.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.638+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:51.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T09:28:51.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.639+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:51.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.639+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:51.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.639+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:51.653 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:51.653 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:51.653 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:51.654 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:51.654 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:51.654 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:51.655 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:51.655 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:51.655 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:51.655 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:51.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.640+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0064730 2021-03-09T09:28:51.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.640+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:51.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.641+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63e0064730 2021-03-09T09:28:51.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.642+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:28:51.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:51.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:51.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:52.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:52.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:52.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:52.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:52.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:52.648+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:53.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:53.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:53.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:53.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:54.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:54.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:54.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:54.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:54.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:54.808+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:28:55.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:55.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:55.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:55.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:56.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:28:56.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:28:56.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6645 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f0013b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:28:56.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6645 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f0013b10).stop 2021-03-09T09:28:56.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:56.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:28:56.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:28:56.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:28:56.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:28:56.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:56.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:28:56.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:56.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:28:56.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:28:56.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:28:56.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:56.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00265f0 con 0x7f63e00070a0 2021-03-09T09:28:56.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:56.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e890 con 0x7f6400145bd0 2021-03-09T09:28:56.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:28:56.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e450 con 0x7f63e005c7c0 2021-03-09T09:28:56.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:28:56.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:28:56.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:56.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:28:56.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.636+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:28:56.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.637+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:28:56.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.637+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:28:56.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.637+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:28:56.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.637+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:28:56.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.637+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:28:56.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.637+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:28:56.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.637+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:28:56.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:28:56.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:56.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:56.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:28:56.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:28:56.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:28:56.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:28:56.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:28:56.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:56.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000fde0 con 0x7f63e005c7c0 2021-03-09T09:28:56.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:28:56.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:28:56.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006aab0 con 0x7f63e005c7c0 2021-03-09T09:28:56.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:28:56.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:56.652 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18029 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:28:56.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:28:56.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:28:56.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.638+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6657 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0654c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:28:56.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0217a0 con 0x7f63e005c7c0 2021-03-09T09:28:56.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.642+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:28:56.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.642+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:28:56.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.642+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:28:56.655 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:28:56.655 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:28:56.655 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:56.656 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:28:56.656 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:28:56.656 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:28:56.656 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:28:56.656 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:28:56.657 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:28:56.657 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:28:56.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.646+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e005c7c0 2021-03-09T09:28:56.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.646+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:28:56.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.646+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:28:56.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.646+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:28:56.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:56.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:56.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:57.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:57.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:57.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:57.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:58.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:58.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:58.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:58.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:59.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:59.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:28:59.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:59.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:28:59.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:28:59.809+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:00.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:00.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:00.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:00.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:01.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.638+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:01.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.638+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:01.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.638+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6657 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0654c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:01.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6657 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0654c0).stop 2021-03-09T09:29:01.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:01.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:01.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:01.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:01.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:29:01.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:01.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:01.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:01.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:01.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:01.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:01.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:01.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f9f0 con 0x7f63e00070a0 2021-03-09T09:29:01.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:01.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400145bd0 2021-03-09T09:29:01.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:01.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066a00 con 0x7f63e0064730 2021-03-09T09:29:01.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:01.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:01.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:01.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:01.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:01.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:01.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:29:01.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:01.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.639+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:01.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.640+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:01.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.640+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:01.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.640+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:01.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.640+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:01.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:01.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:01.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:01.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:01.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:29:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:29:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0021ea0 con 0x7f63e0064730 2021-03-09T09:29:01.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000fde0 con 0x7f63e0064730 2021-03-09T09:29:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:29:01.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:01.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6669 cs=0 l=1 rev1=1 rx=0x7f63fc003fd0 tx=0x7f63fc051850).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:01.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:29:01.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:01.654 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18068 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:01.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0064730 2021-03-09T09:29:01.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:01.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:01.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:01.656 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:01.656 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:01.658 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:01.658 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03c170 con 0x7f63e0064730 2021-03-09T09:29:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.641+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:29:01.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.643+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02bbd0 con 0x7f63e0064730 2021-03-09T09:29:01.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:01.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:01.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:02.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:02.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:02.648+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:03.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:03.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:03.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:03.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:04.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:04.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:04.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:04.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:04.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:04.810+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:05.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:05.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:05.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:05.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:06.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.641+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:06.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.641+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:06.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.641+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6669 cs=0 l=1 rev1=1 rx=0x7f63fc003fd0 tx=0x7f63fc051850).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:06.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.641+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6669 cs=0 l=1 rev1=1 rx=0x7f63fc003fd0 tx=0x7f63fc051850).stop 2021-03-09T09:29:06.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.641+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:06.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.641+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:06.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.641+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:06.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:06.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:29:06.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:06.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:06.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:06.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:06.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:06.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:06.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:06.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00265f0 con 0x7f63e00070a0 2021-03-09T09:29:06.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:06.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e890 con 0x7f6400145bd0 2021-03-09T09:29:06.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:06.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e450 con 0x7f63e005c7c0 2021-03-09T09:29:06.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:06.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:06.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:06.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:06.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.642+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:06.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.643+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:06.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.643+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:29:06.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.643+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:06.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.643+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:06.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:06.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:06.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:06.652 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18106 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:06.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:06.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:06.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:06.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:06.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:06.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:29:06.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:29:06.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:06.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:06.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000fde0 con 0x7f63e005c7c0 2021-03-09T09:29:06.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:06.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:06.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0030400 con 0x7f63e005c7c0 2021-03-09T09:29:06.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:29:06.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:06.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6687 cs=0 l=1 rev1=1 rx=0x7f63f0030e00 tx=0x7f63f0006c00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:06.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:29:06.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.644+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:06.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.646+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e005c7c0 2021-03-09T09:29:06.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.646+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:06.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.646+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:06.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.646+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:06.659 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:06.659 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:06.659 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:06.660 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:06.660 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:06.660 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:06.660 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:06.660 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:06.661 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:06.661 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:06.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.646+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004dc10 con 0x7f63e005c7c0 2021-03-09T09:29:06.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.646+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:06.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.646+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:29:06.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0003b10 con 0x7f63e005c7c0 2021-03-09T09:29:06.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:06.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:06.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:07.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:07.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:07.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:07.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:08.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:08.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:08.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:08.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:09.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:09.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:09.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:09.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:09.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:09.811+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:10.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:10.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:10.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:10.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:11.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:11.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:11.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6687 cs=0 l=1 rev1=1 rx=0x7f63f0030e00 tx=0x7f63f0006c00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:11.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6687 cs=0 l=1 rev1=1 rx=0x7f63f0030e00 tx=0x7f63f0006c00).stop 2021-03-09T09:29:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:11.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:29:11.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:11.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:11.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:11.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:11.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:11.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:11.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:11.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e00070a0 2021-03-09T09:29:11.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:11.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400145bd0 2021-03-09T09:29:11.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:11.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077d60 con 0x7f63e0064730 2021-03-09T09:29:11.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.644+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:11.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.645+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:11.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.645+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:11.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.645+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:11.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.645+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:11.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.645+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:11.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.645+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:29:11.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.645+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:11.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.645+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:11.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:11.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:11.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:11.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:11.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:11.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:11.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:11.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:11.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:29:11.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:29:11.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:11.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:11.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019b80 con 0x7f63e0064730 2021-03-09T09:29:11.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:11.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:11.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001de00 con 0x7f63e0064730 2021-03-09T09:29:11.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:29:11.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.646+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:11.660 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:11.647+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18145 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.648+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6702 cs=0 l=1 rev1=1 rx=0x7f63f4062780 tx=0x7f63f4003bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.648+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0064730 2021-03-09T09:29:11.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.648+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.649+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0064730 2021-03-09T09:29:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.649+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.649+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.649+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:11.664 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:11.664 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:11.664 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:11.664 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:11.665 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0064730 2021-03-09T09:29:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.652+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005b70 con 0x7f63e0064730 2021-03-09T09:29:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:11.652+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:29:12.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:12.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:12.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:12.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:12.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:12.650+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:13.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:13.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:13.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:13.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:14.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:14.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:14.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:14.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:14.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:14.812+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:15.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:15.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:15.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:15.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:16.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:16.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:16.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6702 cs=0 l=1 rev1=1 rx=0x7f63f4062780 tx=0x7f63f4003bb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:16.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 secure :-1 s=READY pgs=6702 cs=0 l=1 rev1=1 rx=0x7f63f4062780 tx=0x7f63f4003bb0).stop 2021-03-09T09:29:16.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:16.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:16.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:16.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:16.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:29:16.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:16.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:16.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:16.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:16.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00653b0 con 0x7f63e00070a0 2021-03-09T09:29:16.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:16.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e890 con 0x7f6400145bd0 2021-03-09T09:29:16.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:16.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00265f0 con 0x7f63e005c7c0 2021-03-09T09:29:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:16.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.647+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:16.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.648+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:16.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.648+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:29:16.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.648+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:16.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.648+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:16.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.648+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:16.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.648+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:16.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.648+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:29:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:29:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:16.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001de00 con 0x7f63e005c7c0 2021-03-09T09:29:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019b80 con 0x7f63e005c7c0 2021-03-09T09:29:16.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:29:16.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:16.662 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:16.649+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18183 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:16.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.650+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6713 cs=0 l=1 rev1=1 rx=0x7f63f4072cd0 tx=0x7f63f400efc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:16.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.650+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:29:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.650+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.651+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072d40 con 0x7f63e005c7c0 2021-03-09T09:29:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.651+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.651+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.651+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:16.665 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:16.667 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:16.667 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:16.667 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:16.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.651+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072d40 con 0x7f63e005c7c0 2021-03-09T09:29:16.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.651+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:16.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.651+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:29:16.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:16.653+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022290 con 0x7f63e005c7c0 2021-03-09T09:29:17.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:17.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:17.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:17.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:18.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:18.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:18.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:18.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:19.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:19.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:19.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:19.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:19.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:19.813+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:20.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:20.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:20.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:20.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:21.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:21.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:21.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6713 cs=0 l=1 rev1=1 rx=0x7f63f4072cd0 tx=0x7f63f400efc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:21.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 secure :-1 s=READY pgs=6713 cs=0 l=1 rev1=1 rx=0x7f63f4072cd0 tx=0x7f63f400efc0).stop 2021-03-09T09:29:21.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:21.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:21.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e007c430 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:21.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e007c430 unknown :-1 s=CLOSED pgs=6713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:21.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:29:21.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:21.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0064730 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f6400145bd0 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63e00070a0 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008a90 con 0x7f63e00070a0 2021-03-09T09:29:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400145bd0 2021-03-09T09:29:21.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001310 con 0x7f63e0064730 2021-03-09T09:29:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:21.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:21.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:21.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0064730 auth_method 0 2021-03-09T09:29:21.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:21.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.650+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:21.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:21.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:21.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:21.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient(hunting): tick 2021-03-09T09:29:21.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 monclient(hunting): continuing hunt 2021-03-09T09:29:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T09:29:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T09:29:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0064730 0x7f63e00219b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 msgr2=0x7f63e0062230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f6400145bd0 0x7f63e0062230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 msgr2=0x7f63e0013620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63e00070a0 0x7f63e0013620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:29:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63d8005850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:21.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63d8008290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63d800acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:21.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:21.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008a90 con 0x7f63d8003400 2021-03-09T09:29:21.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:21.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8008840 2021-03-09T09:29:21.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:21.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001310 con 0x7f63d8005e00 2021-03-09T09:29:21.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f63ecff9700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:21.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.651+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63d800acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63d800acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63d8005850 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63d8005850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63d8008290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:29:21.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:21.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:21.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:21.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:21.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:21.669 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:21.652+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18222 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:21.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:21.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63d800acd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:21.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63d800acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:21.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63d8005850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:21.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63d8005850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:21.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:29:21.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:29:21.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1.5 2021-03-09T09:29:21.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:21.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012fb0 con 0x7f63d8005e00 2021-03-09T09:29:21.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:21.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:21.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0021ea0 con 0x7f63d8005e00 2021-03-09T09:29:21.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:29:21.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:21.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.653+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63d8008290 secure :-1 s=READY pgs=6731 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc071280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:21.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.654+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:29:21.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.654+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:21.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.655+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02cee0 con 0x7f63d8005e00 2021-03-09T09:29:21.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.655+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:21.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.655+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:21.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.655+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:21.675 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:21.675 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:21.675 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:21.676 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:21.676 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:21.676 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:21.676 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:21.677 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:21.677 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:21.677 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:21.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02cee0 con 0x7f63d8005e00 2021-03-09T09:29:21.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.656+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:21.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.656+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:29:21.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:21.657+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63d8005e00 2021-03-09T09:29:24.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:24.814+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:26.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63d8008290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:29:26.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63d8008290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:26.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63d8008290 secure :-1 s=READY pgs=6731 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc071280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:26.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63d8008290 secure :-1 s=READY pgs=6731 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc071280).stop 2021-03-09T09:29:26.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:26.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:26.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63d8008290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:26.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63d8008290 unknown :-1 s=CLOSED pgs=6731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:26.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:29:26.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:26.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:26.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.653+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:26.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:26.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0021ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:26.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:26.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:26.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00132d0 con 0x7f63d8003400 2021-03-09T09:29:26.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:26.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a340 con 0x7f63d8008840 2021-03-09T09:29:26.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:26.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00265f0 con 0x7f63e005c7c0 2021-03-09T09:29:26.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:26.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:26.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:26.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:26.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00219b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:26.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0021ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:26.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:29:26.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:26.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.654+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:26.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:26.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:26.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:26.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:26.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:26.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:26.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:26.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00219b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:26.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:29:26.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:29:26.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1.5 2021-03-09T09:29:26.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:26.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000da30 con 0x7f63e005c7c0 2021-03-09T09:29:26.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:26.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:26.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000fde0 con 0x7f63e005c7c0 2021-03-09T09:29:26.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:29:26.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:26.669 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18260 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:26.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0021ea0 secure :-1 s=READY pgs=6740 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:26.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:29:26.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.655+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:26.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0003d20 con 0x7f63e005c7c0 2021-03-09T09:29:26.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.656+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:26.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.656+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:26.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.656+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:26.672 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:26.672 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:26.672 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:26.672 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:26.673 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:26.673 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:26.673 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:26.673 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:26.674 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:26.674 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:26.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002d730 con 0x7f63e005c7c0 2021-03-09T09:29:26.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.656+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:26.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.656+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:29:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:26.659+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d730 con 0x7f63e005c7c0 2021-03-09T09:29:28.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:28.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:28.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:28.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:28.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:28.402+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:29.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:29.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:29.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:29.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:29.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:29.815+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:30.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:30.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:30.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:30.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:31.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:31.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:31.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0021ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:31.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0021ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:31.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0021ea0 secure :-1 s=READY pgs=6740 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:31.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0021ea0 secure :-1 s=READY pgs=6740 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f004d180).stop 2021-03-09T09:29:31.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:31.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:31.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e0021ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e0021ea0 unknown :-1 s=CLOSED pgs=6740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:29:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:31.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:31.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:31.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e890 con 0x7f63d8003400 2021-03-09T09:29:31.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:31.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000dec0 con 0x7f63d8008840 2021-03-09T09:29:31.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:31.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:29:31.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:31.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:31.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00219b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:31.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:31.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.656+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:31.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.657+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:31.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.657+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:29:31.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.657+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:31.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.657+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:31.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.658+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:31.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.658+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:31.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.658+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:31.672 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:31.658+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18299 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:31.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:31.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:31.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:31.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:31.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00219b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:31.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:29:31.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:29:31.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:31.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:31.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:29:31.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:31.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:31.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012fb0 con 0x7f63d8005e00 2021-03-09T09:29:31.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:29:31.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:31.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6758 cs=0 l=1 rev1=1 rx=0x7f63e000c9e0 tx=0x7f63fc06f290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:31.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:29:31.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.659+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:31.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.660+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:29:31.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.660+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:31.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.660+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:31.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.660+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:31.684 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:31.685 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:31.685 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:31.686 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:31.686 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:31.687 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:31.687 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:31.687 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:31.688 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:31.688 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:31.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:29:31.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.661+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:31.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.661+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:29:31.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:31.663+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63d8005e00 2021-03-09T09:29:32.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:32.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:32.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:32.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:33.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:33.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:33.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:33.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:34.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:34.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:34.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:34.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:34.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:34.816+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:35.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:35.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:35.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:35.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:36.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.402+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:36.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.402+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:36.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.658+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:36.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.658+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:36.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.658+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6758 cs=0 l=1 rev1=1 rx=0x7f63e000c9e0 tx=0x7f63fc06f290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:36.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.658+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6758 cs=0 l=1 rev1=1 rx=0x7f63e000c9e0 tx=0x7f63fc06f290).stop 2021-03-09T09:29:36.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:36.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:36.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:36.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=6758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:36.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:29:36.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:36.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:36.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:36.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:36.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:36.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:36.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:36.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:29:36.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:36.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00292e0 con 0x7f63d8008840 2021-03-09T09:29:36.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:36.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023060 con 0x7f63e005c7c0 2021-03-09T09:29:36.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:36.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:36.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:36.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:36.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.659+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:36.668 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:36.661+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18337 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:36.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:36.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.662+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:29:36.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.662+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:36.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.662+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:36.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.666+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:36.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.666+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:36.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.666+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:36.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:36.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:36.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:36.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:36.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:36.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:29:36.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:29:36.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:36.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:36.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019bb0 con 0x7f63e005c7c0 2021-03-09T09:29:36.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:36.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:36.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000da30 con 0x7f63e005c7c0 2021-03-09T09:29:36.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:29:36.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:36.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6777 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f0054380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:36.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:29:36.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.667+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:36.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001ce80 con 0x7f63e005c7c0 2021-03-09T09:29:36.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.670+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:36.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.670+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:36.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.670+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:36.677 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:36.677 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:36.677 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:36.677 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:36.678 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:36.678 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:36.678 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:36.678 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:36.678 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:36.679 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:36.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004c260 con 0x7f63e005c7c0 2021-03-09T09:29:36.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.670+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:36.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.670+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:29:36.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:36.671+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004c3d0 con 0x7f63e005c7c0 2021-03-09T09:29:37.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:37.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:37.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:37.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:38.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:38.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:38.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:38.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:38.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:38.403+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:39.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:39.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:39.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:39.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:39.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:39.817+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:40.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:40.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:40.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:40.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.403+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:41.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.403+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:41.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.661+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:41.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.661+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:41.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6777 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f0054380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:41.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6777 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f0054380).stop 2021-03-09T09:29:41.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:41.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:41.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:41.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:41.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:29:41.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:41.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:41.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:41.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:41.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:41.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:41.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:41.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:29:41.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:41.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000dec0 con 0x7f63d8008840 2021-03-09T09:29:41.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:41.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8005e00 2021-03-09T09:29:41.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:41.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:41.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:41.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:41.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.662+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:41.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.663+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:41.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.663+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:29:41.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.663+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:41.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.663+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:41.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:41.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:41.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:41.673 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18374 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:41.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:41.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:41.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:41.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:41.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:41.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:29:41.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:29:41.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:41.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:41.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d870 con 0x7f63d8005e00 2021-03-09T09:29:41.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:41.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:41.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019bb0 con 0x7f63d8005e00 2021-03-09T09:29:41.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:29:41.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.664+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:41.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.665+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:29:41.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.665+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:41.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.665+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6788 cs=0 l=1 rev1=1 rx=0x7f63f000df20 tx=0x7f63f001cb80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:41.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.669+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007bc80 con 0x7f63d8005e00 2021-03-09T09:29:41.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.669+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.669+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.669+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:41.680 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:41.681 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:41.682 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:41.682 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:41.682 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:41.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.671+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007baa0 con 0x7f63d8005e00 2021-03-09T09:29:41.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.671+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:41.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.671+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004c470 con 0x7f63d8005e00 2021-03-09T09:29:41.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:41.671+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:29:42.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:42.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:42.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:42.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:43.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:43.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:43.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:43.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:44.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:44.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:44.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:44.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:44.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:44.818+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:45.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:45.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:45.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:45.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:46.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:46.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:46.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:46.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:46.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6788 cs=0 l=1 rev1=1 rx=0x7f63f000df20 tx=0x7f63f001cb80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:46.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6788 cs=0 l=1 rev1=1 rx=0x7f63f000df20 tx=0x7f63f001cb80).stop 2021-03-09T09:29:46.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:46.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:46.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:46.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=6788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:29:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:46.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:46.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:46.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:46.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:46.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001470 con 0x7f63d8003400 2021-03-09T09:29:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f0c0 con 0x7f63d8008840 2021-03-09T09:29:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e005c7c0 2021-03-09T09:29:46.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:46.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:46.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:46.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:46.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.664+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:46.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.665+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:46.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.665+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:29:46.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.665+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.665+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.665+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.665+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.665+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:46.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:46.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:46.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:46.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:46.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:46.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:29:46.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:29:46.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000da30 con 0x7f63e005c7c0 2021-03-09T09:29:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d870 con 0x7f63e005c7c0 2021-03-09T09:29:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:29:46.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:46.679 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18414 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:46.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:29:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.666+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6797 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f006e880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:46.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63e005c7c0 2021-03-09T09:29:46.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.668+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:46.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.668+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:46.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.668+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:46.683 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:46.683 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:46.683 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:46.684 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:46.684 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:46.684 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:46.684 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:46.684 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:46.685 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:46.685 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:46.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028ec0 con 0x7f63e005c7c0 2021-03-09T09:29:46.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.670+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:46.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.670+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:29:46.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:46.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b4c0 con 0x7f63e005c7c0 2021-03-09T09:29:47.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:47.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:47.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:47.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:48.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:48.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:48.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:48.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:48.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:48.404+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:49.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:49.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:49.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:49.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:49.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:49.819+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:50.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:50.404+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:50.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:50.404+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:51.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:29:51.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:51.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6797 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f006e880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:51.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6797 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f006e880).stop 2021-03-09T09:29:51.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:51.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:51.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:51.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:51.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:29:51.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:51.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:51.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:51.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:51.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:51.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:51.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:51.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cfe0 con 0x7f63d8003400 2021-03-09T09:29:51.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:51.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:29:51.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:51.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ec50 con 0x7f63d8005e00 2021-03-09T09:29:51.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:51.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:51.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:51.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:51.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.667+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:51.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.668+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:51.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.668+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:29:51.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.668+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:51.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.668+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:51.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:51.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:51.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:51.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:51.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:51.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:51.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:51.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:51.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:29:51.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:29:51.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:51.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:51.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024ee0 con 0x7f63d8005e00 2021-03-09T09:29:51.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:51.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:51.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001de00 con 0x7f63d8005e00 2021-03-09T09:29:51.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:29:51.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:51.681 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:51.669+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18453 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:51.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.670+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6813 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:51.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.670+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:29:51.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.670+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:51.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.671+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f63d8005e00 2021-03-09T09:29:51.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.671+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:51.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.671+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:51.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.671+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:51.684 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:51.684 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:51.684 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:51.684 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:51.685 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:51.685 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:51.685 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:51.685 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:51.685 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:51.686 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:51.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f63d8005e00 2021-03-09T09:29:51.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.672+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:51.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.672+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:29:51.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:51.674+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4072cb0 con 0x7f63d8005e00 2021-03-09T09:29:52.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:52.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:52.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:52.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:53.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:53.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:53.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:53.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:54.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:54.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:54.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:54.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:54.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:54.820+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:29:55.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:55.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:55.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:55.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:56.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:56.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:56.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:29:56.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:29:56.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6813 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:29:56.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6813 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4005110).stop 2021-03-09T09:29:56.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:56.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:29:56.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:29:56.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=6813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:29:56.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:29:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:29:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:29:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:29:56.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:29:56.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:56.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:29:56.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:56.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:29:56.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:29:56.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013860 con 0x7f63e005c7c0 2021-03-09T09:29:56.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:29:56.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:29:56.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:56.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:29:56.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.670+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:29:56.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.671+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:29:56.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.671+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:29:56.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.671+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:29:56.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.671+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:29:56.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.671+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:29:56.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.671+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:29:56.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.671+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:29:56.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:29:56.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:56.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:56.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:29:56.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:29:56.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:29:56.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:29:56.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:56.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:56.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006db00 con 0x7f63e005c7c0 2021-03-09T09:29:56.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:29:56.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:29:56.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000da30 con 0x7f63e005c7c0 2021-03-09T09:29:56.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:29:56.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:56.686 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:29:56.672+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18491 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:29:56.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.673+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6826 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0006680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:29:56.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.673+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:29:56.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.673+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:29:56.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.675+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63e005c7c0 2021-03-09T09:29:56.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.675+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:29:56.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.675+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:29:56.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.675+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:29:56.688 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:29:56.689 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:29:56.689 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:56.689 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:29:56.689 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:29:56.690 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:29:56.690 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:29:56.690 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:29:56.690 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:29:56.690 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:29:56.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028ec0 con 0x7f63e005c7c0 2021-03-09T09:29:56.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.677+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:29:56.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.677+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:29:56.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:56.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001ccb0 con 0x7f63e005c7c0 2021-03-09T09:29:57.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:57.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:57.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:57.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:58.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:58.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:58.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:58.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:58.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:58.406+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:29:59.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:59.405+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:29:59.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:59.405+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:29:59.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:29:59.821+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:00.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:00.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:00.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:00.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:01.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:01.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6826 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0006680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6826 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0006680).stop 2021-03-09T09:30:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:30:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:01.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:01.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:01.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:01.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:30:01.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:01.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000dd60 con 0x7f63d8008840 2021-03-09T09:30:01.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:01.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8005e00 2021-03-09T09:30:01.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:01.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:01.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:01.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:01.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.673+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:01.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.674+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:01.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.674+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:30:01.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.674+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:01.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.674+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:01.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.674+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:01.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.674+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:01.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.674+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:01.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.674+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:01.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:01.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:01.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:01.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:01.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:30:01.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:30:01.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:01.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:01.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e840 con 0x7f63d8005e00 2021-03-09T09:30:01.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:01.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:01.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024ee0 con 0x7f63d8005e00 2021-03-09T09:30:01.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:30:01.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:01.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6837 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4046ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:01.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:30:01.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:01.691 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:01.675+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18539 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:01.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.678+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40224a0 con 0x7f63d8005e00 2021-03-09T09:30:01.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.678+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:01.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.678+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:01.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.678+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:01.693 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:01.693 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:01.693 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:01.693 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:01.694 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:01.694 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:01.694 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:01.694 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:01.694 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:01.695 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:01.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63d8005e00 2021-03-09T09:30:01.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.682+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:01.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.682+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:30:01.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:01.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63d8005e00 2021-03-09T09:30:02.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:02.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:02.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:02.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:03.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:03.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:03.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:03.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:04.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:04.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:04.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:04.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:04.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:04.822+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:05.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:05.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:05.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:05.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:06.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.406+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:06.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.406+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:06.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:06.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:06.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6837 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4046ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:06.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6837 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4046ef0).stop 2021-03-09T09:30:06.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=6837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:30:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:06.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:06.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:06.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:06.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000dd60 con 0x7f63d8003400 2021-03-09T09:30:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013820 con 0x7f63d8008840 2021-03-09T09:30:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63e005c7c0 2021-03-09T09:30:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:06.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.676+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:06.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.677+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:06.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.677+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:06.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.677+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:30:06.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.677+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:06.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.677+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:06.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.677+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:06.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.677+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:06.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.677+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:06.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:06.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:30:06.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:30:06.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:06.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:06.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e110 con 0x7f63e005c7c0 2021-03-09T09:30:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:30:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:30:06.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:06.692 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:06.678+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18577 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.679+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:30:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.679+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.679+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6850 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc046bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.681+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:30:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.681+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.681+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.681+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:06.695 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:06.695 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:06.695 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:06.695 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:06.695 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:06.696 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:06.696 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:06.696 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:06.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d6d0 con 0x7f63e005c7c0 2021-03-09T09:30:06.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.685+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.685+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:30:06.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:06.686+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:30:07.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:07.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:07.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:07.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:08.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:08.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:08.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:08.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:08.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:08.407+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:09.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:09.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:09.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:09.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:09.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:09.823+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:10.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:10.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:10.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:10.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:11.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:11.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6850 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc046bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6850 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc046bc0).stop 2021-03-09T09:30:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:30:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:11.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:11.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:11.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:11.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:30:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000dd60 con 0x7f63d8008840 2021-03-09T09:30:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012e30 con 0x7f63d8005e00 2021-03-09T09:30:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:11.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.679+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.680+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.680+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:30:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.680+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.680+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.680+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.680+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.680+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:30:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:30:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:30:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:11.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:11.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e840 con 0x7f63d8005e00 2021-03-09T09:30:11.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:30:11.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:11.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6866 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4017e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:11.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:30:11.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:11.694 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:11.681+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18616 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:11.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.683+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4061dc0 con 0x7f63d8005e00 2021-03-09T09:30:11.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.683+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:11.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.683+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:11.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.683+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:11.696 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:11.696 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:11.696 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:11.697 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:11.697 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:11.697 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:11.697 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:11.697 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:11.698 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:11.698 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:11.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.683+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63d8005e00 2021-03-09T09:30:11.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.683+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:11.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.683+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:30:11.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:11.684+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022290 con 0x7f63d8005e00 2021-03-09T09:30:12.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:12.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:12.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:12.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:13.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:13.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:13.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:13.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:14.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:14.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:14.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:14.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:14.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:14.824+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:15.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:15.407+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:15.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:15.407+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:16.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:16.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:16.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:30:16.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:16.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6866 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4017e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:16.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6866 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4017e60).stop 2021-03-09T09:30:16.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:16.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:16.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:16.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=6866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:16.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:30:16.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:16.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:16.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:16.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:16.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:16.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:16.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:16.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068520 con 0x7f63d8003400 2021-03-09T09:30:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066670 con 0x7f63d8008840 2021-03-09T09:30:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e005c7c0 2021-03-09T09:30:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.682+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.683+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.683+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.683+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.683+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.683+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:16.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.683+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:30:16.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.683+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:16.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.683+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:16.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:16.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:16.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:16.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:16.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:16.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:16.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:16.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:16.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:30:16.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:30:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0077a90 con 0x7f63e005c7c0 2021-03-09T09:30:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e110 con 0x7f63e005c7c0 2021-03-09T09:30:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:30:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:16.697 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:16.684+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18652 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:16.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.685+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6882 cs=0 l=1 rev1=1 rx=0x7f63e002fab0 tx=0x7f63fc075fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:16.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.685+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:30:16.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.685+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:16.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03f450 con 0x7f63e005c7c0 2021-03-09T09:30:16.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.687+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:16.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.687+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.687+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:16.701 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:16.701 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:16.701 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:16.701 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:16.701 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:16.702 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:16.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03f280 con 0x7f63e005c7c0 2021-03-09T09:30:16.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.687+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:16.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.687+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:30:16.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:16.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:30:17.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:17.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:17.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:17.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:18.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:18.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:18.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:18.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:18.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:18.408+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:19.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:19.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:19.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:19.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:19.825+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:20.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:20.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:20.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:20.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:21.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:21.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:21.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6882 cs=0 l=1 rev1=1 rx=0x7f63e002fab0 tx=0x7f63fc075fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:21.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6882 cs=0 l=1 rev1=1 rx=0x7f63e002fab0 tx=0x7f63fc075fe0).stop 2021-03-09T09:30:21.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:21.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:21.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:21.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:21.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:30:21.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:21.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:21.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:21.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:21.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:21.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:21.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:21.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00658d0 con 0x7f63d8003400 2021-03-09T09:30:21.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:21.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8008840 2021-03-09T09:30:21.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:21.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000dd60 con 0x7f63d8005e00 2021-03-09T09:30:21.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:21.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:21.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:21.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:21.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:21.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:21.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:30:21.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:21.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:21.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:21.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:21.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.685+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:21.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:21.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:21.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:21.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:21.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:21.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:30:21.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:30:21.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:21.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:21.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c780 con 0x7f63d8005e00 2021-03-09T09:30:21.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:21.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:21.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:30:21.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:30:21.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.686+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:21.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.687+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6891 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f402df30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:21.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.687+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:30:21.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.687+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:21.701 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:21.687+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18693 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:21.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63d8005e00 2021-03-09T09:30:21.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.688+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:21.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.688+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:21.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.688+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:21.703 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:21.704 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:21.704 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:21.704 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:21.705 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:21.705 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:21.705 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:21.706 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:21.706 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:21.706 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:21.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.689+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63d8005e00 2021-03-09T09:30:21.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.689+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:21.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.689+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:30:21.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:21.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4046be0 con 0x7f63d8005e00 2021-03-09T09:30:22.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:22.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:22.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:22.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:23.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:23.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:23.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:23.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:24.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:24.408+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:24.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:24.408+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:24.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:24.826+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:25.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:25.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:25.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:25.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:26.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:26.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:26.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:30:26.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:26.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6891 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f402df30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:26.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6891 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f402df30).stop 2021-03-09T09:30:26.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:26.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:26.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:26.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=6891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:26.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:30:26.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:26.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:26.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:26.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:26.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:26.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:26.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:26.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ce80 con 0x7f63d8003400 2021-03-09T09:30:26.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:26.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:30:26.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:26.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63e005c7c0 2021-03-09T09:30:26.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:26.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:26.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:26.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:26.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:26.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:26.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:30:26.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:26.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.688+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:26.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.689+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:26.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.689+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:26.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.689+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:26.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.689+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:26.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.689+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:26.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:26.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:26.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:26.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:30:26.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:30:26.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:26.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:26.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e005c7c0 2021-03-09T09:30:26.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:26.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:26.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0077a90 con 0x7f63e005c7c0 2021-03-09T09:30:26.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:30:26.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:26.703 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18729 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:26.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.690+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6908 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:26.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.691+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:30:26.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.691+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:26.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.692+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023a70 con 0x7f63e005c7c0 2021-03-09T09:30:26.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.692+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.692+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.692+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:26.707 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023840 con 0x7f63e005c7c0 2021-03-09T09:30:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.696+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.696+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:30:26.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:26.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63e005c7c0 2021-03-09T09:30:27.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:27.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:27.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:27.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:28.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:28.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:28.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:28.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:28.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:28.410+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:29.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:29.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:29.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:29.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:29.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:29.827+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:30.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:30.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:30.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:30.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:31.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.409+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:31.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.409+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:31.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:30:31.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:31.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6908 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:31.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6908 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc009990).stop 2021-03-09T09:30:31.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:31.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:31.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:31.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:31.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:30:31.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:31.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:31.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:31.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:31.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:31.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:31.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:31.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00151c0 con 0x7f63d8003400 2021-03-09T09:30:31.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:31.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cd40 con 0x7f63d8008840 2021-03-09T09:30:31.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:31.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000dd60 con 0x7f63d8005e00 2021-03-09T09:30:31.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:31.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:31.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:31.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:31.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:31.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:31.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:30:31.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:31.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.691+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:31.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.692+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:31.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.692+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:31.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.692+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:31.702 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18777 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:31.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:31.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:31.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:31.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:31.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:31.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:30:31.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:30:31.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:31.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:31.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024ee0 con 0x7f63d8005e00 2021-03-09T09:30:31.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:31.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:31.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:30:31.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:30:31.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.693+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:31.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.694+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:30:31.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6924 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc021330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:31.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.694+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:31.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8005e00 2021-03-09T09:30:31.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.697+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:31.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.697+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:31.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.697+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:31.709 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:31.709 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:31.709 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:31.709 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:31.710 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:31.710 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:31.710 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:31.710 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:31.711 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:31.711 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:31.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0653a0 con 0x7f63d8005e00 2021-03-09T09:30:31.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.697+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:31.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.698+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc008070 con 0x7f63d8005e00 2021-03-09T09:30:31.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:31.698+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:30:32.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:32.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:32.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:32.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:33.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:33.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:33.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:33.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:34.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:34.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:34.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:34.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:34.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:34.828+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:35.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:35.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:35.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:35.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:36.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:36.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:36.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:36.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:36.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6924 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc021330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:36.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6924 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc021330).stop 2021-03-09T09:30:36.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:36.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:36.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:36.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=6924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:36.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:30:36.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:36.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:36.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:36.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:36.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:36.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:36.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:36.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:30:36.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:36.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8008840 2021-03-09T09:30:36.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:36.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008c90 con 0x7f63e005c7c0 2021-03-09T09:30:36.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:36.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:36.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:36.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:36.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.694+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:36.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:36.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:30:36.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:36.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:36.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:36.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:36.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:36.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:36.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:36.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:36.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:36.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:36.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:30:36.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:30:36.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:36.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:36.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e005c7c0 2021-03-09T09:30:36.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:36.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:36.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006db00 con 0x7f63e005c7c0 2021-03-09T09:30:36.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:30:36.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.695+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:36.709 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:36.696+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18818 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:36.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.696+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:30:36.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.696+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:36.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.696+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6933 cs=0 l=1 rev1=1 rx=0x7f63e00280f0 tx=0x7f63f000ef60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:36.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.698+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e005c7c0 2021-03-09T09:30:36.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.698+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:36.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.698+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:36.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.698+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:36.711 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:36.711 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:36.712 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:36.712 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:36.712 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:36.712 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:36.712 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:36.713 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:36.713 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:36.713 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:36.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.698+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f63e005c7c0 2021-03-09T09:30:36.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.698+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:36.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.698+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:30:36.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:36.699+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f006ed50 con 0x7f63e005c7c0 2021-03-09T09:30:37.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:37.410+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:37.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:37.410+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:38.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:38.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:38.411+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:39.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:39.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:39.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:39.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:39.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:39.829+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:40.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:40.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:40.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:40.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:41.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:41.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:41.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:30:41.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:41.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6933 cs=0 l=1 rev1=1 rx=0x7f63e00280f0 tx=0x7f63f000ef60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:41.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6933 cs=0 l=1 rev1=1 rx=0x7f63e00280f0 tx=0x7f63f000ef60).stop 2021-03-09T09:30:41.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:41.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:41.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:41.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:41.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:30:41.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:41.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:41.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:41.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:41.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:41.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:41.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:41.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000dd60 con 0x7f63d8003400 2021-03-09T09:30:41.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:41.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8008840 2021-03-09T09:30:41.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:41.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:30:41.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:41.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:41.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:41.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:41.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.697+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:41.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.698+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:41.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.698+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:30:41.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.698+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:41.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.698+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:41.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.699+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:41.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.699+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:41.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.699+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:41.710 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:41.699+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18857 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:41.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:41.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:41.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:41.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:41.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:41.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:30:41.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:30:41.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:41.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:41.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e440 con 0x7f63d8005e00 2021-03-09T09:30:41.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:41.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:41.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:30:41.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:30:41.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.700+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:41.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.701+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6956 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0030e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:41.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.701+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:30:41.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.701+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:41.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.705+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T09:30:41.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.705+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:41.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.705+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:41.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.705+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:41.716 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:41.716 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:41.717 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:41.717 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:41.717 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:41.717 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:41.718 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:41.718 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:41.718 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:41.719 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:41.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.705+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T09:30:41.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.705+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:41.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.705+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:30:41.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:41.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007a9d0 con 0x7f63d8005e00 2021-03-09T09:30:42.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:42.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:42.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:42.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:43.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:43.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:43.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:43.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:44.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:44.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:44.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:44.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:44.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:44.830+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:45.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:45.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:45.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:45.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:46.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.411+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:46.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.411+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:46.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6956 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0030e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6956 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0030e80).stop 2021-03-09T09:30:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:46.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=6956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:30:46.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:46.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:46.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:46.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:46.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:46.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:46.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:46.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cba0 con 0x7f63d8003400 2021-03-09T09:30:46.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:46.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc60 con 0x7f63d8008840 2021-03-09T09:30:46.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:46.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027000 con 0x7f63e005c7c0 2021-03-09T09:30:46.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:46.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:46.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.700+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.701+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.701+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:30:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.701+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:46.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.701+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.701+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.701+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:46.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.701+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:46.712 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18902 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:46.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:46.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:46.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:30:46.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:30:46.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:46.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:46.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.702+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005cd80 con 0x7f63e005c7c0 2021-03-09T09:30:46.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.703+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:46.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.703+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:46.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.703+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c780 con 0x7f63e005c7c0 2021-03-09T09:30:46.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.703+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:30:46.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.703+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:46.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.703+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6965 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f40036d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:46.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.703+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:30:46.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.703+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:46.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.705+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022290 con 0x7f63e005c7c0 2021-03-09T09:30:46.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.705+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:46.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.705+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:46.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.705+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:46.719 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:46.719 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:46.719 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:46.719 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:46.720 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:46.720 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:46.720 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:46.720 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:46.720 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:46.721 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:46.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63e005c7c0 2021-03-09T09:30:46.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.706+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:46.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.706+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:30:46.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:46.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022290 con 0x7f63e005c7c0 2021-03-09T09:30:47.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:47.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:47.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:48.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:48.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:48.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:48.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:48.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:48.412+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:49.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:49.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:49.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:49.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:49.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:49.831+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:50.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:50.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:50.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:50.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:51.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:51.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:51.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:30:51.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:51.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6965 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f40036d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:51.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6965 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f40036d0).stop 2021-03-09T09:30:51.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:51.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:51.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:51.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:51.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:30:51.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:51.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:51.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:51.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:51.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:51.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:51.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:51.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008c90 con 0x7f63d8003400 2021-03-09T09:30:51.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:51.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e1d0 con 0x7f63d8008840 2021-03-09T09:30:51.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:51.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017890 con 0x7f63d8005e00 2021-03-09T09:30:51.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:51.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:51.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:51.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:51.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:30:51.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:51.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:51.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:51.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:51.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.703+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:51.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:51.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:51.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:51.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:51.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:51.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:51.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:51.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:30:51.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:30:51.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:51.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:51.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000dd10 con 0x7f63d8005e00 2021-03-09T09:30:51.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:51.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:51.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024ee0 con 0x7f63d8005e00 2021-03-09T09:30:51.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:30:51.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.704+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:51.723 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:51.705+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18941 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:51.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.705+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6976 cs=0 l=1 rev1=1 rx=0x7f63fc04a830 tx=0x7f63fc023ef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:51.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.705+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:30:51.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.705+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:51.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63d8005e00 2021-03-09T09:30:51.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.706+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:51.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.706+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:51.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.706+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:51.726 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:51.726 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:51.726 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:51.727 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:51.727 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:51.727 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:51.727 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:51.728 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:51.728 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:51.728 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:51.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d510 con 0x7f63d8005e00 2021-03-09T09:30:51.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.706+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.706+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:30:51.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:51.707+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T09:30:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:52.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:52.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:53.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:53.412+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:53.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:53.412+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:54.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:54.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:54.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:54.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:54.832+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:30:55.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:55.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:55.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:55.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:56.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:56.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:56.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:30:56.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:30:56.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6976 cs=0 l=1 rev1=1 rx=0x7f63fc04a830 tx=0x7f63fc023ef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:30:56.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=6976 cs=0 l=1 rev1=1 rx=0x7f63fc04a830 tx=0x7f63fc023ef0).stop 2021-03-09T09:30:56.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:56.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:30:56.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:30:56.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=6976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:30:56.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:30:56.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:56.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:30:56.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:56.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:30:56.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:30:56.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:30:56.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:56.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:30:56.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:56.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63d8008840 2021-03-09T09:30:56.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:30:56.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077840 con 0x7f63e005c7c0 2021-03-09T09:30:56.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:30:56.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:30:56.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:30:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.706+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:30:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:30:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:30:56.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:30:56.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:30:56.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:30:56.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:30:56.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:30:56.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:30:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:30:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:30:56.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:30:56.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:30:56.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:56.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:56.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067990 con 0x7f63e005c7c0 2021-03-09T09:30:56.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:30:56.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:30:56.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e440 con 0x7f63e005c7c0 2021-03-09T09:30:56.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:30:56.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.707+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:56.720 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:30:56.708+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 18979 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:30:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.708+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:30:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6990 cs=0 l=1 rev1=1 rx=0x7f63e002f9d0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:30:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.708+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:30:56.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.710+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030ce0 con 0x7f63e005c7c0 2021-03-09T09:30:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.710+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:30:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.710+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:30:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.710+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:30:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:30:56.722 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:30:56.723 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:56.723 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:30:56.723 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:30:56.723 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:30:56.723 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:30:56.724 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:30:56.724 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:30:56.724 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:30:56.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.710+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e005c7c0 2021-03-09T09:30:56.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.710+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:30:56.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.710+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:30:56.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:56.710+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001cbf0 con 0x7f63e005c7c0 2021-03-09T09:30:57.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:57.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:57.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:57.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:58.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:58.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:58.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:58.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:58.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:58.414+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:30:59.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:59.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:30:59.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:59.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:30:59.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:30:59.833+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:00.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:00.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:00.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:01.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:01.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:01.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6990 cs=0 l=1 rev1=1 rx=0x7f63e002f9d0 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=6990 cs=0 l=1 rev1=1 rx=0x7f63e002f9d0 tx=0x7f63f004db20).stop 2021-03-09T09:31:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:01.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:01.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:01.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=6990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:01.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:31:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:01.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:01.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:01.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c150 con 0x7f63d8003400 2021-03-09T09:31:01.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:01.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008c90 con 0x7f63d8008840 2021-03-09T09:31:01.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:01.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e1d0 con 0x7f63d8005e00 2021-03-09T09:31:01.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.708+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:01.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.709+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:01.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.709+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:01.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.709+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:01.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.709+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:01.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.709+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:01.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.709+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:31:01.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.709+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:01.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.709+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:01.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:01.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:01.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:01.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:01.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:01.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:01.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:01.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:01.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:31:01.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:31:01.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:01.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:01.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028770 con 0x7f63d8005e00 2021-03-09T09:31:01.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:01.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:01.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067990 con 0x7f63d8005e00 2021-03-09T09:31:01.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:31:01.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.710+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:01.725 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:01.711+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19018 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:01.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.711+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7004 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0030e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:01.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.711+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:31:01.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.711+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:01.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.712+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T09:31:01.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.712+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:01.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.712+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:01.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.712+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:01.727 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:01.727 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:01.728 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:01.728 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:01.728 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:01.728 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:01.728 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:01.729 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:01.729 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:01.729 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:01.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.712+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T09:31:01.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.712+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:01.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.713+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:31:01.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:01.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004c470 con 0x7f63d8005e00 2021-03-09T09:31:02.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:02.413+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:02.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:02.413+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:03.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:03.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:03.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:03.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:04.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:04.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:04.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:04.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:04.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:04.834+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:05.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:05.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:05.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:05.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:06.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:06.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:06.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:06.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:06.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7004 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0030e10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:06.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7004 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0030e10).stop 2021-03-09T09:31:06.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:06.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:06.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:06.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:06.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:31:06.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:06.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e005c7c0 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:06.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:06.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:06.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:06.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:06.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:06.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:31:06.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:06.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63d8008840 2021-03-09T09:31:06.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:06.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e1d0 con 0x7f63e005c7c0 2021-03-09T09:31:06.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:06.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:06.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:06.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:06.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:06.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:06.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:06.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:06.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.712+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:06.729 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:06.714+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19056 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:06.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:06.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T09:31:06.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:06.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T09:31:06.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:06.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e005c7c0 auth_method 0 2021-03-09T09:31:06.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:06.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:06.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:06.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:06.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:06.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:06.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:06.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:06.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:06.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:06.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:31:06.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:31:06.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:06.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:06.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001cb50 con 0x7f63e005c7c0 2021-03-09T09:31:06.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:06.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:06.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005cd80 con 0x7f63e005c7c0 2021-03-09T09:31:06.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:31:06.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:06.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=7027 cs=0 l=1 rev1=1 rx=0x7f63f4073070 tx=0x7f63f4072cd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:06.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e005c7c0 2021-03-09T09:31:06.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:06.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e005c7c0 2021-03-09T09:31:06.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:06.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:06.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:06.929 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:06.929 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:06.929 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:06.929 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:06.930 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:06.930 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:06.930 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:06.930 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:06.930 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:06.931 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:06.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T09:31:06.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40224c0 con 0x7f63e005c7c0 2021-03-09T09:31:06.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:06.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.913+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:31:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=7027 cs=0 l=1 rev1=1 rx=0x7f63f4073070 tx=0x7f63f4072cd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 secure :-1 s=READY pgs=7027 cs=0 l=1 rev1=1 rx=0x7f63f4073070 tx=0x7f63f4072cd0).stop 2021-03-09T09:31:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e005c7c0 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:31:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:06.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:06.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:06.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c150 con 0x7f63d8003400 2021-03-09T09:31:06.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:06.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008c90 con 0x7f63d8008840 2021-03-09T09:31:06.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:06.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a9b0 con 0x7f63d8005e00 2021-03-09T09:31:06.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:31:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:06.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:06.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:06.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:06.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:06.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:06.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:06.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:06.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:31:06.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:31:06.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:06.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:06.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005cd80 con 0x7f63d8005e00 2021-03-09T09:31:06.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:06.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:06.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000dd10 con 0x7f63d8005e00 2021-03-09T09:31:06.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:31:06.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.914+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:06.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:31:06.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:06.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7028 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc06f290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:06.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:31:06.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:06.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:06.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:06.945 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:06.945 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:06.945 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:06.946 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:06.946 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:06.946 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:06.946 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:06.946 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:06.947 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:06.947 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:06.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63d8005e00 2021-03-09T09:31:06.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:06.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.915+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:31:06.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:06.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T09:31:07.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:07.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:07.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:07.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:08.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:08.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:08.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:08.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:08.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:08.415+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:09.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:09.414+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:09.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:09.414+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:09.835+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:10.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:10.415+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:10.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:10.415+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:11.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:11.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:11.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:11.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:11.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7028 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc06f290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:11.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7028 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc06f290).stop 2021-03-09T09:31:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:11.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:31:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:31:11.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:11.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63d8008840 2021-03-09T09:31:11.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:11.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c150 con 0x7f63e0018390 2021-03-09T09:31:11.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:11.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:11.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.715+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:11.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:31:11.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:11.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:11.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:11.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:11.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:11.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:11.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:11.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:11.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:11.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:11.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:31:11.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:31:11.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:11.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:11.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0021ea0 con 0x7f63e0018390 2021-03-09T09:31:11.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:11.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:11.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005cd80 con 0x7f63e0018390 2021-03-09T09:31:11.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:31:11.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.716+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:11.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.717+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:31:11.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.717+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:11.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.717+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7031 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc046b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:11.732 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:11.717+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19095 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:11.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63e0018390 2021-03-09T09:31:11.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.718+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:11.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.718+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:11.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.718+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:11.734 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:11.734 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:11.734 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:11.735 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:11.735 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:11.735 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:11.735 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:11.735 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:11.736 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:11.736 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:11.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0653a0 con 0x7f63e0018390 2021-03-09T09:31:11.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.718+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:11.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.718+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:31:11.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:11.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63e0018390 2021-03-09T09:31:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:12.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:12.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:13.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:13.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:13.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:13.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:14.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:14.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:14.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:14.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:14.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:14.836+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:15.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:15.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:15.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:16.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:16.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:16.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:16.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:16.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7031 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc046b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:16.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7031 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc046b50).stop 2021-03-09T09:31:16.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:16.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:16.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:16.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7031 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:16.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:31:16.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:16.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:16.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:16.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:16.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:16.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:16.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:16.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008c90 con 0x7f63d8003400 2021-03-09T09:31:16.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:16.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e1d0 con 0x7f63d8008840 2021-03-09T09:31:16.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:16.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:31:16.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:16.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:16.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.718+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:16.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.719+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:16.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.719+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:16.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:16.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:31:16.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:16.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:16.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:16.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:16.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:16.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:16.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:16.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:16.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:16.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:16.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:31:16.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:31:16.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:16.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:16.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d30 con 0x7f63d8005e00 2021-03-09T09:31:16.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:16.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:16.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0021ea0 con 0x7f63d8005e00 2021-03-09T09:31:16.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:31:16.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.720+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:16.736 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:16.721+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19133 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:16.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.722+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7049 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:16.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.722+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:31:16.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.722+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:16.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.725+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058af0 con 0x7f63d8005e00 2021-03-09T09:31:16.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.725+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:16.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.725+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:16.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.725+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:16.739 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:16.739 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:16.739 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:16.740 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:16.740 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:16.740 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:16.740 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:16.740 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:16.741 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:16.741 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:16.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.725+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8005e00 2021-03-09T09:31:16.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.725+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:16.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.725+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02bbd0 con 0x7f63d8005e00 2021-03-09T09:31:16.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:16.725+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:31:17.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:17.416+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:17.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:17.416+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:18.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:18.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:18.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:18.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:18.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:18.417+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:19.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:19.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:19.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:19.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:19.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:19.837+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:20.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:20.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:20.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:20.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:21.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:21.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:21.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:21.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:21.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7049 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:21.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7049 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a740).stop 2021-03-09T09:31:21.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:21.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:21.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:21.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7049 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:21.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:31:21.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:21.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:21.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:21.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:21.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:21.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:21.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:21.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63d8003400 2021-03-09T09:31:21.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:21.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bde0 con 0x7f63d8008840 2021-03-09T09:31:21.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:21.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:31:21.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.721+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:21.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:21.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:21.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:21.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:21.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:21.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:31:21.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:21.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:21.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:21.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:21.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:21.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.722+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:21.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:21.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:21.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:21.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:21.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:31:21.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:31:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00175a0 con 0x7f63e0018390 2021-03-09T09:31:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:21.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:21.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028770 con 0x7f63e0018390 2021-03-09T09:31:21.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:31:21.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:21.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:31:21.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:21.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.723+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7057 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:21.736 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:21.724+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19182 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:21.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.724+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T09:31:21.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.724+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:21.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.724+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:21.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.724+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:21.738 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:21.738 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:21.738 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:21.738 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:21.739 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:21.740 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.724+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:31:21.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.724+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:21.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.724+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:31:21.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:21.726+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T09:31:22.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:22.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:22.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:22.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:23.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:23.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:23.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:23.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:24.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:24.417+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:24.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:24.417+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:24.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:24.838+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:25.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:25.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:25.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:25.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:26.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:26.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:26.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7057 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:26.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7057 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006680).stop 2021-03-09T09:31:26.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:26.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:26.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:26.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:26.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:31:26.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:26.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:26.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:26.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:26.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:26.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:26.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:26.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e1d0 con 0x7f63d8003400 2021-03-09T09:31:26.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:26.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bde0 con 0x7f63d8008840 2021-03-09T09:31:26.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:26.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068ce0 con 0x7f63d8005e00 2021-03-09T09:31:26.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:26.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:26.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.724+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:26.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:26.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:26.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:26.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:31:26.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:26.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:26.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:26.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:26.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:26.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:26.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.725+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:26.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:26.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:26.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:26.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:31:26.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:31:26.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:26.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005cd80 con 0x7f63d8005e00 2021-03-09T09:31:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002d30 con 0x7f63d8005e00 2021-03-09T09:31:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:31:26.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:26.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:31:26.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:26.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.726+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7072 cs=0 l=1 rev1=1 rx=0x7f63e002f9d0 tx=0x7f63fc04d510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:26.739 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:26.727+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19220 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:26.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03ceb0 con 0x7f63d8005e00 2021-03-09T09:31:26.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.727+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:26.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.727+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:26.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.727+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:26.741 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:26.742 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:26.742 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:26.742 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:26.742 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:26.743 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:26.743 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:26.743 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:26.743 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:26.743 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:26.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051eb0 con 0x7f63d8005e00 2021-03-09T09:31:26.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.727+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:26.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.727+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:31:26.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.729+0000 7f6406553700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T09:31:26.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.729+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:26.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:26.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.729+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc05dbc0 con 0x7f63d8005e00 2021-03-09T09:31:26.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7072 cs=0 l=1 rev1=1 rx=0x7f63e002f9d0 tx=0x7f63fc04d510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:26.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7072 cs=0 l=1 rev1=1 rx=0x7f63e002f9d0 tx=0x7f63fc04d510).stop 2021-03-09T09:31:26.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:26.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:26.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:26.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:26.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:31:26.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:26.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:26.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:26.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:26.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:26.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:26.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:26.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:31:26.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:26.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63d8008840 2021-03-09T09:31:26.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:26.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bde0 con 0x7f63e0018390 2021-03-09T09:31:26.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:26.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:26.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.730+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:26.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.731+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:26.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.731+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:26.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.731+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:26.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.731+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:31:26.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.731+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:26.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.731+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:26.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.731+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:26.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.731+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:26.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.731+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:26.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:26.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:26.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:26.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:26.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:26.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:31:26.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:31:26.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:26.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:26.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00777f0 con 0x7f63e0018390 2021-03-09T09:31:26.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:26.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:26.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00175a0 con 0x7f63e0018390 2021-03-09T09:31:26.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:31:26.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:26.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:31:26.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:26.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.732+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067cc0 secure :-1 s=READY pgs=7084 cs=0 l=1 rev1=1 rx=0x7f63f004ce80 tx=0x7f63f004cb00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:26.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.734+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:31:26.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.734+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:26.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.734+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:26.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.734+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:26.756 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:26.757 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:26.757 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:26.757 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:26.757 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:26.757 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:26.758 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:26.758 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:26.758 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:26.758 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:26.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.734+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:31:26.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.734+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:26.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.734+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:31:26.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:26.736+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T09:31:27.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:27.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:27.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:27.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:28.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:28.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:28.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:28.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:28.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:28.418+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:29.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:29.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:29.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:29.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:29.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:29.839+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:30.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:30.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:30.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:30.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:31.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:31.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:31.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.727+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0067cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:31.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.727+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0067cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:31.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.727+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067cc0 secure :-1 s=READY pgs=7084 cs=0 l=1 rev1=1 rx=0x7f63f004ce80 tx=0x7f63f004cb00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:31.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.727+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067cc0 secure :-1 s=READY pgs=7084 cs=0 l=1 rev1=1 rx=0x7f63f004ce80 tx=0x7f63f004cb00).stop 2021-03-09T09:31:31.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.727+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:31.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.727+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:31.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0067cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:31.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067cc0 unknown :-1 s=CLOSED pgs=7084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:31.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:31:31.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:31.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:31.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:31.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:31.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:31.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:31.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:31.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001cd0 con 0x7f63d8003400 2021-03-09T09:31:31.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:31.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00653b0 con 0x7f63d8008840 2021-03-09T09:31:31.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:31.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00658d0 con 0x7f63d8005e00 2021-03-09T09:31:31.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:31.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:31.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:31.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:31.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:31:31.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:31.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.728+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:31.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.729+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:31.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.729+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:31.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.729+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:31.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.729+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:31.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.729+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:31.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:31.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:31.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:31.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:31.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:31.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:31:31.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:31:31.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:31.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:31.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:31:31.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:31.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:31.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00777f0 con 0x7f63d8005e00 2021-03-09T09:31:31.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:31:31.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:31.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:31:31.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:31.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7089 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f007eee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:31.743 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:31.730+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19259 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:31.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.736+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007ea80 con 0x7f63d8005e00 2021-03-09T09:31:31.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.736+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:31.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.736+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:31.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.736+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:31.744 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:31.744 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:31.745 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:31.745 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:31.745 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:31.745 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:31.745 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:31.746 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:31.746 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:31.746 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:31.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.736+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005df00 con 0x7f63d8005e00 2021-03-09T09:31:31.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.736+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:31.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.736+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:31:31.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:31.736+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f006e520 con 0x7f63d8005e00 2021-03-09T09:31:32.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:32.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:32.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:32.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:33.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:33.418+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:33.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:33.418+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:34.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:34.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:34.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:34.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:34.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:34.840+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:35.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:35.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:35.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:35.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:36.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:36.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:36.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:36.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:36.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7089 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f007eee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:36.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7089 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f007eee0).stop 2021-03-09T09:31:36.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:36.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:36.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:36.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:36.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:31:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e1d0 con 0x7f63d8003400 2021-03-09T09:31:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:31:36.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:36.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00658d0 con 0x7f63e0018390 2021-03-09T09:31:36.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.730+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:36.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.731+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:36.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.731+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:36.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.731+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:36.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.731+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:36.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.731+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:36.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.731+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:31:36.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.731+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:36.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.731+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:36.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:36.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:36.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:36.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:36.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:36.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:36.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:36.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:36.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:31:36.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:31:36.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:36.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:36.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028770 con 0x7f63e0018390 2021-03-09T09:31:36.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:36.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:36.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.732+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001cb50 con 0x7f63e0018390 2021-03-09T09:31:36.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.733+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:31:36.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.733+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:36.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.733+0000 7f6405551700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T09:31:36.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.733+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:36.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.733+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:36.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.733+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0x7f63f4034d40 tx=0x7f63f40058b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:36.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.733+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:36.746 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:36.733+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19295 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:36.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient: get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:31:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 0 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-03-09T09:31:36.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:36.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:36.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:31:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:36.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bde0 con 0x7f63d8003400 2021-03-09T09:31:36.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:36.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:31:36.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:36.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0078100 con 0x7f63d8005e00 2021-03-09T09:31:36.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:36.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:36.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:36.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:36.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:31:36.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:36.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:36.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:36.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:36.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:36.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:36.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:36.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:36.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:36.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:36.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:36.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:36.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:31:36.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:31:36.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:36.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:36.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00777f0 con 0x7f63d8005e00 2021-03-09T09:31:36.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:36.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:36.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028770 con 0x7f63d8005e00 2021-03-09T09:31:36.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:31:36.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:36.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:31:36.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7112 cs=0 l=1 rev1=1 rx=0x7f63f4003c40 tx=0x7f63f40189c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:36.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:36.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4018e20 con 0x7f63d8005e00 2021-03-09T09:31:36.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:36.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:36.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:36.958 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:36.958 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:36.958 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:36.959 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:36.959 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:36.959 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:36.959 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:36.960 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:36.960 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:36.960 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:36.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4006a90 con 0x7f63d8005e00 2021-03-09T09:31:36.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:36.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.935+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:31:36.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:36.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400b790 con 0x7f63d8005e00 2021-03-09T09:31:37.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:37.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:37.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:37.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:38.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:38.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:38.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:38.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:38.419+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:39.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:39.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:39.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:39.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:39.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:39.841+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:40.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:40.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:40.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:40.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:41.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:41.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:41.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.732+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:41.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.732+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:41.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.732+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7112 cs=0 l=1 rev1=1 rx=0x7f63f4003c40 tx=0x7f63f40189c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:41.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.732+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7112 cs=0 l=1 rev1=1 rx=0x7f63f4003c40 tx=0x7f63f40189c0).stop 2021-03-09T09:31:41.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:41.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:41.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:41.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:41.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:31:41.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:41.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:41.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:41.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:41.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:41.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:41.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:41.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:31:41.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:41.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019710 con 0x7f63d8008840 2021-03-09T09:31:41.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:41.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000da80 con 0x7f63e0018390 2021-03-09T09:31:41.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:41.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:41.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:41.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:41.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:41.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:41.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:31:41.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:41.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:41.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:41.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:41.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.733+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:41.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:41.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:41.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:41.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:41.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:41.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:31:41.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:31:41.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:41.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:41.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028770 con 0x7f63e0018390 2021-03-09T09:31:41.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:41.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:41.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T09:31:41.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:31:41.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.734+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:41.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7115 cs=0 l=1 rev1=1 rx=0x7f63f0029000 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:41.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:31:41.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:41.761 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19337 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:41.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T09:31:41.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:41.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:41.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:41.764 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:41.764 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:41.765 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:41.765 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:41.766 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:41.766 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:41.767 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:41.767 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:41.767 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:41.768 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:41.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T09:31:41.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:41.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.735+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:31:41.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:41.737+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007bdb0 con 0x7f63e0018390 2021-03-09T09:31:42.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:42.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:42.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:42.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:43.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:43.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:43.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:43.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:44.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:44.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:44.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:44.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:44.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:44.841+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:45.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:45.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:45.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:45.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:46.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:46.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:46.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:31:46.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:46.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7115 cs=0 l=1 rev1=1 rx=0x7f63f0029000 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:46.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7115 cs=0 l=1 rev1=1 rx=0x7f63f0029000 tx=0x7f63f0000c30).stop 2021-03-09T09:31:46.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:31:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:46.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:46.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:46.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:46.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:46.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:46.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:46.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d050 con 0x7f63d8003400 2021-03-09T09:31:46.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:46.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8008840 2021-03-09T09:31:46.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:46.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017a90 con 0x7f63d8005e00 2021-03-09T09:31:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.735+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.736+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.736+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:46.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.736+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:46.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.736+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:31:46.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.736+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:46.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.736+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:46.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.736+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:46.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.736+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:46.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.736+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:46.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:46.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:46.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:46.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:46.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:46.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:31:46.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:31:46.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019000 con 0x7f63d8005e00 2021-03-09T09:31:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028770 con 0x7f63d8005e00 2021-03-09T09:31:46.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:31:46.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:46.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c7d0 secure :-1 s=READY pgs=7128 cs=0 l=1 rev1=1 rx=0x7f63f002fc90 tx=0x7f63f005dd20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:46.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:31:46.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.737+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:46.753 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:46.738+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19376 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:46.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.739+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T09:31:46.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.739+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:46.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.739+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:46.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.739+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:46.756 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:46.756 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:46.756 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:46.756 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:46.756 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:46.757 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:46.757 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:46.757 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:46.757 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:46.758 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:46.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.739+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T09:31:46.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.739+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:46.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.739+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:31:46.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:46.740+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002f480 con 0x7f63d8005e00 2021-03-09T09:31:47.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:47.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:47.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:47.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:48.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:48.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:48.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:48.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:48.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:48.419+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:49.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:49.419+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:49.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:49.419+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:49.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:49.842+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:50.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:50.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:50.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:50.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:51.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:51.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:31:51.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:51.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c7d0 secure :-1 s=READY pgs=7128 cs=0 l=1 rev1=1 rx=0x7f63f002fc90 tx=0x7f63f005dd20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:51.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c7d0 secure :-1 s=READY pgs=7128 cs=0 l=1 rev1=1 rx=0x7f63f002fc90 tx=0x7f63f005dd20).stop 2021-03-09T09:31:51.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001c7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c7d0 unknown :-1 s=CLOSED pgs=7128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:31:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:51.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:51.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:51.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:51.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bde0 con 0x7f63d8003400 2021-03-09T09:31:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:31:51.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:51.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e0018390 2021-03-09T09:31:51.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:51.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:51.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:51.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:51.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.738+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:51.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.739+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:51.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.739+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:31:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.739+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.739+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.739+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.739+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.739+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:51.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:51.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:51.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:51.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:51.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:51.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:31:51.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:31:51.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:51.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:51.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0077660 con 0x7f63e0018390 2021-03-09T09:31:51.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019000 con 0x7f63e0018390 2021-03-09T09:31:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:31:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.740+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.741+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7143 cs=0 l=1 rev1=1 rx=0x7f63e0030be0 tx=0x7f63f0027f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:51.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.741+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:31:51.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.741+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:51.753 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:51.741+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19404 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:51.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040a60 con 0x7f63e0018390 2021-03-09T09:31:51.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.742+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:51.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.742+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:51.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.742+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:51.755 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:51.755 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:51.756 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:51.756 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:51.756 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:51.756 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:51.757 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:51.757 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:51.757 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:51.757 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:51.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T09:31:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.743+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.743+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:31:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:51.744+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002f480 con 0x7f63e0018390 2021-03-09T09:31:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:52.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:52.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:53.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:53.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:53.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:53.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:54.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:54.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:54.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:54.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:54.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:54.843+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:31:55.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:55.420+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:55.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:55.420+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:56.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:56.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.741+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:31:56.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.741+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:31:56.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.741+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7143 cs=0 l=1 rev1=1 rx=0x7f63e0030be0 tx=0x7f63f0027f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:31:56.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.741+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7143 cs=0 l=1 rev1=1 rx=0x7f63e0030be0 tx=0x7f63f0027f90).stop 2021-03-09T09:31:56.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.741+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:56.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.741+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:31:56.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.741+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:31:56.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.741+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:31:56.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:31:56.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:56.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:31:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:31:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:31:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:31:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f740 con 0x7f63d8003400 2021-03-09T09:31:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d0b0 con 0x7f63d8008840 2021-03-09T09:31:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:31:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023cb0 con 0x7f63d8005e00 2021-03-09T09:31:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:31:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:31:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.742+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:31:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:31:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:31:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:31:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:31:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:31:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:31:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:31:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:31:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:31:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:56.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:31:56.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.743+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:31:56.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:31:56.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:31:56.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:56.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:56.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028770 con 0x7f63d8005e00 2021-03-09T09:31:56.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:31:56.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:31:56.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0077660 con 0x7f63d8005e00 2021-03-09T09:31:56.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:31:56.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:56.761 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19455 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:31:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7157 cs=0 l=1 rev1=1 rx=0x7f63f000f040 tx=0x7f63f000efd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:31:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:31:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.744+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:31:56.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.746+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007fd40 con 0x7f63d8005e00 2021-03-09T09:31:56.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.746+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:31:56.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.746+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:31:56.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.746+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:31:56.763 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:31:56.764 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:31:56.764 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:56.764 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:31:56.764 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:31:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:31:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:31:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:31:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:31:56.765 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:31:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:31:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.747+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:31:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.747+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:31:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:56.748+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002f480 con 0x7f63d8005e00 2021-03-09T09:31:57.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:57.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:57.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:57.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:58.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:58.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:58.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:58.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:58.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:58.421+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:31:59.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:59.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:31:59.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:59.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:31:59.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:31:59.844+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:00.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:00.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:00.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:00.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:01.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:01.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:01.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.744+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:01.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.744+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:01.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.744+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7157 cs=0 l=1 rev1=1 rx=0x7f63f000f040 tx=0x7f63f000efd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:01.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.744+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7157 cs=0 l=1 rev1=1 rx=0x7f63f000f040 tx=0x7f63f000efd0).stop 2021-03-09T09:32:01.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:01.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:01.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:01.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:01.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:32:01.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:01.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:01.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:01.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:01.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:01.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:01.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:01.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e0c0 con 0x7f63d8003400 2021-03-09T09:32:01.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:01.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f690 con 0x7f63d8008840 2021-03-09T09:32:01.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:01.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f040 con 0x7f63e0018390 2021-03-09T09:32:01.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:01.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:01.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:01.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:01.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:01.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:01.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:32:01.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:01.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:01.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:01.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:01.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.745+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:01.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:01.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:01.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:01.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:01.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:01.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:32:01.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:32:01.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:01.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:01.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a8a0 con 0x7f63e0018390 2021-03-09T09:32:01.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:01.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:01.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005cd80 con 0x7f63e0018390 2021-03-09T09:32:01.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:32:01.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.746+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:01.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.747+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:32:01.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.747+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:01.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.747+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7171 cs=0 l=1 rev1=1 rx=0x7f63e0003380 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:01.761 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:01.747+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19492 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:01.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.749+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e0018390 2021-03-09T09:32:01.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.749+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.749+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.749+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:01.764 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:01.764 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:01.764 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:01.765 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:01.765 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:01.765 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:01.765 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:01.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.749+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e0018390 2021-03-09T09:32:01.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.749+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:01.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.749+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:32:01.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:01.750+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63e0018390 2021-03-09T09:32:02.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:02.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:02.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:02.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:03.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:03.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:03.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:03.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:04.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:04.421+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:04.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:04.421+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:04.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:04.845+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:05.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:05.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:05.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:05.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:06.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:06.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:06.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:06.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:06.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7171 cs=0 l=1 rev1=1 rx=0x7f63e0003380 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:06.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7171 cs=0 l=1 rev1=1 rx=0x7f63e0003380 tx=0x7f63fc009990).stop 2021-03-09T09:32:06.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:06.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:06.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:06.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:06.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:32:06.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:06.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:06.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:06.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:06.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:06.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:06.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:06.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:32:06.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:06.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:32:06.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:06.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001230 con 0x7f63d8005e00 2021-03-09T09:32:06.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:06.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:06.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:06.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:06.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:06.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:06.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:32:06.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:06.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.748+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:06.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:06.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:06.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:06.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:06.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:06.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:06.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:06.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:06.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:32:06.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:32:06.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:06.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:06.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006a5c0 con 0x7f63d8005e00 2021-03-09T09:32:06.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:06.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:06.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00777f0 con 0x7f63d8005e00 2021-03-09T09:32:06.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:32:06.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.749+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:06.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.750+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7184 cs=0 l=1 rev1=1 rx=0x7f63f404a010 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:06.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.750+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:32:06.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.750+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:06.764 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:06.750+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19530 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:06.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4018e20 con 0x7f63d8005e00 2021-03-09T09:32:06.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.751+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:06.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.751+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:06.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.751+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:06.765 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:06.767 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:06.767 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:06.767 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:06.767 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:06.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.752+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4006a90 con 0x7f63d8005e00 2021-03-09T09:32:06.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.752+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:06.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.752+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:32:06.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:06.753+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8005e00 2021-03-09T09:32:07.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:07.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:07.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:07.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:08.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:08.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:08.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:08.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:08.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:08.422+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:09.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:09.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:09.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:09.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:09.846+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:10.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:10.422+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:10.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:10.422+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:11.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:11.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:11.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:11.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:11.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7184 cs=0 l=1 rev1=1 rx=0x7f63f404a010 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:11.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7184 cs=0 l=1 rev1=1 rx=0x7f63f404a010 tx=0x7f63f4017ed0).stop 2021-03-09T09:32:11.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:11.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:11.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:11.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:11.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:32:11.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:11.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:11.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:11.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:11.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:11.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:11.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:11.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000efa0 con 0x7f63d8003400 2021-03-09T09:32:11.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:11.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066910 con 0x7f63d8008840 2021-03-09T09:32:11.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:11.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f690 con 0x7f63e0018390 2021-03-09T09:32:11.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:11.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:11.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:11.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:11.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.751+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:11.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.752+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:11.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.752+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:32:11.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.752+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:11.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.752+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:11.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.752+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:11.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.752+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:11.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.752+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:11.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:11.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:11.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:11.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:11.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:11.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:32:11.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:32:11.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:11.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:11.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a910 con 0x7f63e0018390 2021-03-09T09:32:11.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:11.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:11.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a8a0 con 0x7f63e0018390 2021-03-09T09:32:11.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:32:11.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:11.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00114d0 secure :-1 s=READY pgs=7198 cs=0 l=1 rev1=1 rx=0x7f63e001ba70 tx=0x7f63fc052f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:11.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:32:11.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.753+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:11.767 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:11.754+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19569 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:11.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.755+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc033970 con 0x7f63e0018390 2021-03-09T09:32:11.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.755+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:11.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.755+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:11.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.755+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:11.768 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:11.769 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:11.769 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:11.770 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:11.770 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:11.770 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:11.770 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:11.771 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:11.771 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:11.771 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:11.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.755+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0018390 2021-03-09T09:32:11.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.755+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:11.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.755+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:32:11.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:11.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63e0018390 2021-03-09T09:32:12.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:12.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:12.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:12.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:13.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:13.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:13.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:13.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:14.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:14.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:14.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:14.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:14.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:14.847+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:15.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:15.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:15.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:15.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:16.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:16.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:16.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:16.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:16.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00114d0 secure :-1 s=READY pgs=7198 cs=0 l=1 rev1=1 rx=0x7f63e001ba70 tx=0x7f63fc052f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:16.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00114d0 secure :-1 s=READY pgs=7198 cs=0 l=1 rev1=1 rx=0x7f63e001ba70 tx=0x7f63fc052f70).stop 2021-03-09T09:32:16.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:16.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:16.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00114d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:16.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00114d0 unknown :-1 s=CLOSED pgs=7198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:16.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:32:16.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:16.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:16.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:16.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:16.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:16.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:16.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:16.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cf0 con 0x7f63d8003400 2021-03-09T09:32:16.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:16.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:32:16.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:16.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025340 con 0x7f63d8005e00 2021-03-09T09:32:16.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.754+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:16.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.755+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:16.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.755+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:16.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.755+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:16.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.756+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:32:16.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.756+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:16.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.756+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:16.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.756+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:16.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.756+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:16.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.756+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:16.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.756+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:16.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.756+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:16.765 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:16.757+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19609 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:16.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:16.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:16.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:16.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:16.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:16.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:32:16.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:32:16.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:16.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:16.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:32:16.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:16.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:16.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028770 con 0x7f63d8005e00 2021-03-09T09:32:16.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:32:16.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:16.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7220 cs=0 l=1 rev1=1 rx=0x7f63e001b9e0 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:16.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:32:16.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.758+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:16.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.760+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T09:32:16.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.760+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:16.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.760+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:16.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.760+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:16.771 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:16.771 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:16.772 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:16.772 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:16.772 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:16.772 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:16.772 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:16.773 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:16.773 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:16.773 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:16.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.765+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T09:32:16.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.765+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:16.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.765+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007fd40 con 0x7f63d8005e00 2021-03-09T09:32:16.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:16.765+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:32:17.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:17.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:17.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:17.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:18.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:18.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:18.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:18.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:18.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:18.424+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:19.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:19.423+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:19.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:19.423+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:19.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:19.848+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:20.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:20.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:20.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:20.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:21.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:21.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:21.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:32:21.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:21.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7220 cs=0 l=1 rev1=1 rx=0x7f63e001b9e0 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:21.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7220 cs=0 l=1 rev1=1 rx=0x7f63e001b9e0 tx=0x7f63f0000c30).stop 2021-03-09T09:32:21.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:21.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:21.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:21.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:21.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:32:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:21.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:21.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:21.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:21.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:21.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:21.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f740 con 0x7f63d8003400 2021-03-09T09:32:21.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:21.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006dc10 con 0x7f63d8008840 2021-03-09T09:32:21.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:21.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f690 con 0x7f63e0018390 2021-03-09T09:32:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:21.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:21.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.760+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:21.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.760+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:32:21.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.760+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:21.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.760+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:21.776 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:21.760+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19646 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:21.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.762+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:21.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.762+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:21.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.762+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:21.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:21.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:21.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:21.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:21.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:21.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:32:21.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:32:21.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:21.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:21.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028770 con 0x7f63e0018390 2021-03-09T09:32:21.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:21.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:21.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006a5c0 con 0x7f63e0018390 2021-03-09T09:32:21.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:32:21.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:21.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7236 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f404bfb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:32:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.763+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:21.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034be0 con 0x7f63e0018390 2021-03-09T09:32:21.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.764+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:21.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.764+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.764+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:21.789 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:21.789 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:21.790 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:21.790 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:21.790 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:21.792 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.767+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4006a90 con 0x7f63e0018390 2021-03-09T09:32:21.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.767+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:21.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.767+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4034be0 con 0x7f63e0018390 2021-03-09T09:32:21.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:21.767+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:32:22.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:22.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:22.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:22.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:23.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:23.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:23.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:24.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:24.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:24.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:24.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:24.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:24.849+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:25.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:25.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:25.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:25.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:26.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:26.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:26.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:26.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:26.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7236 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f404bfb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:26.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7236 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f404bfb0).stop 2021-03-09T09:32:26.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:26.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:26.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:26.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:26.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:32:26.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:26.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:26.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:26.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:26.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:26.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:26.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:26.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066910 con 0x7f63d8003400 2021-03-09T09:32:26.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:26.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:32:26.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:26.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016420 con 0x7f63d8005e00 2021-03-09T09:32:26.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:26.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:26.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:26.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:26.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:26.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.762+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:26.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:32:26.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:26.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:26.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:26.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:26.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:26.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:26.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:26.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:26.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:26.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:26.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:32:26.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:32:26.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:26.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:26.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d770 con 0x7f63d8005e00 2021-03-09T09:32:26.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:26.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:26.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:32:26.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:32:26.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:26.778 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:26.763+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19686 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:26.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000c240 secure :-1 s=READY pgs=7243 cs=0 l=1 rev1=1 rx=0x7f63f0030fd0 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:26.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.764+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:32:26.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.764+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:26.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001cac0 con 0x7f63d8005e00 2021-03-09T09:32:26.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.766+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:26.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.766+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:26.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.766+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:26.781 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:26.781 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:26.781 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:26.781 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:26.782 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:26.782 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:26.782 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:26.782 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:26.782 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:26.783 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:26.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007bed0 con 0x7f63d8005e00 2021-03-09T09:32:26.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.766+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:26.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f000f810 con 0x7f63d8005e00 2021-03-09T09:32:26.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:26.766+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:32:27.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:27.424+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:27.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:27.424+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:28.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:28.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:28.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:28.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:28.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:28.425+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:29.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:29.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:29.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:29.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:29.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:29.850+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:30.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:30.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:30.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:30.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:31.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:31.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:31.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:31.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:31.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000c240 secure :-1 s=READY pgs=7243 cs=0 l=1 rev1=1 rx=0x7f63f0030fd0 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:31.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000c240 secure :-1 s=READY pgs=7243 cs=0 l=1 rev1=1 rx=0x7f63f0030fd0 tx=0x7f63f0006610).stop 2021-03-09T09:32:31.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:31.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:31.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000c240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:31.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000c240 unknown :-1 s=CLOSED pgs=7243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:31.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:32:31.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:31.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:31.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:31.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:31.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:31.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:31.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:31.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T09:32:31.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:31.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024220 con 0x7f63d8008840 2021-03-09T09:32:31.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:31.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f690 con 0x7f63e0018390 2021-03-09T09:32:31.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:31.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:31.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:31.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:31.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.764+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:31.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.765+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:31.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.765+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:32:31.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.765+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:31.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.765+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:31.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.766+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:31.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.766+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:31.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.766+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:31.775 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:31.766+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19723 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:31.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:31.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:32:31.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:32:31.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:31.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066690 con 0x7f63e0018390 2021-03-09T09:32:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028770 con 0x7f63e0018390 2021-03-09T09:32:31.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:32:31.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.767+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:31.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.768+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7261 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f40098a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:31.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.768+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:32:31.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.768+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b840 con 0x7f63e0018390 2021-03-09T09:32:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.771+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.771+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.771+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:31.781 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:31.782 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:31.783 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:31.783 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b610 con 0x7f63e0018390 2021-03-09T09:32:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.771+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.771+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:32:31.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:31.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0018390 2021-03-09T09:32:32.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:32.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:32.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:32.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:33.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:33.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:33.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:33.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:34.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:34.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:34.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:34.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:34.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:34.851+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:35.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:35.425+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:35.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:35.425+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:36.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:36.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.767+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.767+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:36.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.767+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7261 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f40098a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:36.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.767+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7261 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f40098a0).stop 2021-03-09T09:32:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:36.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:36.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:36.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:32:36.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:36.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:36.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006dc10 con 0x7f63d8003400 2021-03-09T09:32:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:32:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:36.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019050 con 0x7f63d8005e00 2021-03-09T09:32:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:36.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.768+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.769+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.769+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:32:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.769+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:36.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.769+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:36.778 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:36.769+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19763 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:36.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.769+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:36.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.769+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:36.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.769+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:36.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:36.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:36.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:36.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:36.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:36.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:32:36.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:32:36.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:36.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:36.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028770 con 0x7f63d8005e00 2021-03-09T09:32:36.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:36.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:36.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066690 con 0x7f63d8005e00 2021-03-09T09:32:36.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:32:36.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:36.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:32:36.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:36.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.770+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7272 cs=0 l=1 rev1=1 rx=0x7f63f4061d30 tx=0x7f63f4005e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:36.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.773+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4061b70 con 0x7f63d8005e00 2021-03-09T09:32:36.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.773+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:36.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.773+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:36.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.773+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:36.784 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:36.785 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:36.785 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:36.785 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:36.785 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:36.785 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:36.786 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:36.786 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:36.786 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:36.786 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:36.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.773+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63d8005e00 2021-03-09T09:32:36.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.773+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:36.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.773+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:32:36.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:36.773+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4034c70 con 0x7f63d8005e00 2021-03-09T09:32:37.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:37.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:37.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:37.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:38.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:38.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:38.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:38.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:38.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:38.426+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:39.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:39.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:39.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:39.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:39.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:39.852+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:40.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:40.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:40.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:40.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:41.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.426+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:41.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.426+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:41.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:41.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:41.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7272 cs=0 l=1 rev1=1 rx=0x7f63f4061d30 tx=0x7f63f4005e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:41.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7272 cs=0 l=1 rev1=1 rx=0x7f63f4061d30 tx=0x7f63f4005e60).stop 2021-03-09T09:32:41.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:41.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:41.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:41.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:41.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:32:41.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:41.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:41.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:41.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:41.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:41.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:41.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:41.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022e60 con 0x7f63d8003400 2021-03-09T09:32:41.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:41.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00223f0 con 0x7f63d8008840 2021-03-09T09:32:41.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:41.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e0018390 2021-03-09T09:32:41.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:41.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:41.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:41.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:41.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.770+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:41.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.771+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:41.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.771+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:32:41.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.771+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:41.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.771+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:41.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.771+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:41.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.771+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:41.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.771+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:41.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:41.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:41.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:41.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:41.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:41.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:32:41.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:32:41.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:41.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:41.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002dfc0 con 0x7f63e0018390 2021-03-09T09:32:41.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:41.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:41.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d770 con 0x7f63e0018390 2021-03-09T09:32:41.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:32:41.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:41.785 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19808 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:41.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7285 cs=0 l=1 rev1=1 rx=0x7f63f007ade0 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:41.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:32:41.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.772+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:41.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63e0018390 2021-03-09T09:32:41.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.775+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:41.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.775+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:41.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.775+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:41.788 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:41.788 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:41.788 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:41.788 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:41.788 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:41.788 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:41.789 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:41.789 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:41.789 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:41.789 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:41.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007ae10 con 0x7f63e0018390 2021-03-09T09:32:41.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.775+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:41.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.775+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:32:41.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:41.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f000f810 con 0x7f63e0018390 2021-03-09T09:32:42.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:42.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:42.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:42.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:43.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:43.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:43.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:43.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:44.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:44.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:44.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:44.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:44.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:44.853+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:45.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:45.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:45.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:45.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:46.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:46.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:46.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7285 cs=0 l=1 rev1=1 rx=0x7f63f007ade0 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7285 cs=0 l=1 rev1=1 rx=0x7f63f007ade0 tx=0x7f63f0013b80).stop 2021-03-09T09:32:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:46.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:46.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:46.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:46.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:32:46.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:46.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:46.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:46.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:46.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:46.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014000 con 0x7f63d8003400 2021-03-09T09:32:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f690 con 0x7f63d8008840 2021-03-09T09:32:46.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066910 con 0x7f63d8005e00 2021-03-09T09:32:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:46.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:46.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.773+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:46.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.774+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:46.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.774+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:46.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.774+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:32:46.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.774+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:46.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.774+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:46.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:46.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:46.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:46.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:46.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:46.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:46.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:46.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:46.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:32:46.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:32:46.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:46.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:46.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:32:46.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:46.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:46.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028770 con 0x7f63d8005e00 2021-03-09T09:32:46.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:32:46.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.775+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:46.789 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:46.776+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19848 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:46.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.777+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7302 cs=0 l=1 rev1=1 rx=0x7f63f4063270 tx=0x7f63f40701e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:46.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.777+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:32:46.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.777+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:46.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40297d0 con 0x7f63d8005e00 2021-03-09T09:32:46.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.779+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:46.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.779+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:46.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.779+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:46.792 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:46.792 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:46.792 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:46.792 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:46.792 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:46.793 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:46.793 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:46.793 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:46.793 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:46.794 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:46.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.780+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40297d0 con 0x7f63d8005e00 2021-03-09T09:32:46.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.780+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:46.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.780+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:32:46.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:46.781+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4061d60 con 0x7f63d8005e00 2021-03-09T09:32:47.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:47.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:47.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:47.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:48.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:48.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:48.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:48.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:48.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:48.427+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:49.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:49.427+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:49.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:49.427+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:49.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:49.854+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:50.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:50.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:50.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:50.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:51.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:51.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:51.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.776+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:32:51.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.776+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:51.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.776+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7302 cs=0 l=1 rev1=1 rx=0x7f63f4063270 tx=0x7f63f40701e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:51.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.776+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7302 cs=0 l=1 rev1=1 rx=0x7f63f4063270 tx=0x7f63f40701e0).stop 2021-03-09T09:32:51.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:51.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:51.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:51.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:51.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:32:51.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:51.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:51.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0015810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:51.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:51.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:51.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:51.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:51.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:32:51.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:51.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f7f0 con 0x7f63d8008840 2021-03-09T09:32:51.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:51.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e0018390 2021-03-09T09:32:51.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:51.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:51.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:51.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:51.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:51.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0015810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:51.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:32:51.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:51.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.777+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:51.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.778+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:51.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.778+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:51.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.778+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:51.788 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19885 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:51.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:51.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:51.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:51.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:51.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:51.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:32:51.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:32:51.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:51.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:51.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001cb50 con 0x7f63e0018390 2021-03-09T09:32:51.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:51.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:51.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a910 con 0x7f63e0018390 2021-03-09T09:32:51.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:32:51.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.779+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:51.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.780+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0015810 secure :-1 s=READY pgs=7312 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc0039e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:51.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.780+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:32:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.780+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.781+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63e0018390 2021-03-09T09:32:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.781+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.781+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:51.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.781+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:51.794 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:51.796 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.781+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d260 con 0x7f63e0018390 2021-03-09T09:32:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.781+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.781+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:32:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:51.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63e0018390 2021-03-09T09:32:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:52.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:52.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:53.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:53.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:53.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:53.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:54.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:54.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:54.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:54.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:54.855+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:32:55.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:55.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:55.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:55.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:56.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:56.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:56.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0015810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:32:56.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0015810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:32:56.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0015810 secure :-1 s=READY pgs=7312 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc0039e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:32:56.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0015810 secure :-1 s=READY pgs=7312 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc0039e0).stop 2021-03-09T09:32:56.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:56.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:32:56.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0015810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:32:56.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0015810 unknown :-1 s=CLOSED pgs=7312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:32:56.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:32:56.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:56.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:32:56.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00093e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:56.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:32:56.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:32:56.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:32:56.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:56.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025340 con 0x7f63d8003400 2021-03-09T09:32:56.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:56.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066910 con 0x7f63d8008840 2021-03-09T09:32:56.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:32:56.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f690 con 0x7f63d8005e00 2021-03-09T09:32:56.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.779+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:32:56.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.780+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:32:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.780+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:32:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.780+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.780+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:32:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.780+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00093e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:32:56.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.780+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:32:56.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.780+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:32:56.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.780+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:32:56.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.782+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:32:56.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.782+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:32:56.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.782+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:32:56.791 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:32:56.782+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19925 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:32:56.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.782+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:32:56.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:56.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:56.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:32:56.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:32:56.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:32:56.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:32:56.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:56.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:56.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a910 con 0x7f63d8005e00 2021-03-09T09:32:56.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:32:56.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:32:56.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001cb50 con 0x7f63d8005e00 2021-03-09T09:32:56.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:32:56.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:56.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00093e0 secure :-1 s=READY pgs=7333 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc06f290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:32:56.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:32:56.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.783+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:32:56.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00f350 con 0x7f63d8005e00 2021-03-09T09:32:56.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.785+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:32:56.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.785+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:32:56.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.785+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:32:56.798 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:32:56.798 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:32:56.798 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:56.798 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:32:56.799 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:32:56.799 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:32:56.799 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:32:56.799 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:32:56.800 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:32:56.800 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:32:56.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00f350 con 0x7f63d8005e00 2021-03-09T09:32:56.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.787+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:32:56.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.787+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:32:56.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:56.788+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02bbd0 con 0x7f63d8005e00 2021-03-09T09:32:57.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:57.428+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:57.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:57.428+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:58.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:58.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:58.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:58.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:58.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:58.429+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:32:59.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:59.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:32:59.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:59.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:32:59.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:32:59.856+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:00.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:00.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:00.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:00.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:01.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:01.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:01.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:01.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00093e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:01.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00093e0 secure :-1 s=READY pgs=7333 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc06f290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:01.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00093e0 secure :-1 s=READY pgs=7333 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc06f290).stop 2021-03-09T09:33:01.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:01.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:01.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00093e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:01.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00093e0 unknown :-1 s=CLOSED pgs=7333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:01.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:33:01.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:01.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:01.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:01.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:01.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:01.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:01.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:01.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:33:01.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:01.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065e10 con 0x7f63d8008840 2021-03-09T09:33:01.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:01.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010d50 con 0x7f63e0018390 2021-03-09T09:33:01.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.783+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:01.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.784+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:01.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.784+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:01.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.784+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:01.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.784+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:01.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.784+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:01.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.784+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:33:01.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.784+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:01.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.784+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:01.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.785+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:01.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.785+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:01.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.785+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:01.796 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:01.785+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 19962 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:01.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:01.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:01.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:01.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:01.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:01.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:33:01.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:33:01.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:01.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:01.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066760 con 0x7f63e0018390 2021-03-09T09:33:01.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:01.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:01.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002dfc0 con 0x7f63e0018390 2021-03-09T09:33:01.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:33:01.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:01.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7341 cs=0 l=1 rev1=1 rx=0x7f63e002c2d0 tx=0x7f63f0006bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:01.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:33:01.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.786+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:01.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63e0018390 2021-03-09T09:33:01.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.792+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:01.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.792+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:01.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.792+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:01.803 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:01.804 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:01.804 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:01.804 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:01.804 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:01.805 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:01.805 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:01.805 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:01.805 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:01.806 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:01.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00541f0 con 0x7f63e0018390 2021-03-09T09:33:01.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.792+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:01.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.792+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:33:01.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:01.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004cc30 con 0x7f63e0018390 2021-03-09T09:33:02.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:02.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:02.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:02.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:03.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:03.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:03.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:03.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:04.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:04.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:04.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:04.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:04.857+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:05.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:05.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:05.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:05.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:06.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.429+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:06.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.429+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:06.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:06.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:06.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7341 cs=0 l=1 rev1=1 rx=0x7f63e002c2d0 tx=0x7f63f0006bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:06.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7341 cs=0 l=1 rev1=1 rx=0x7f63e002c2d0 tx=0x7f63f0006bc0).stop 2021-03-09T09:33:06.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:06.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:06.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:06.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:06.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:33:06.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:06.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:06.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:06.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:06.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:06.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:06.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:06.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018100 con 0x7f63d8003400 2021-03-09T09:33:06.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:06.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d670 con 0x7f63d8008840 2021-03-09T09:33:06.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:06.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:33:06.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:06.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:06.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:06.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:06.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:06.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.787+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:06.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.787+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:33:06.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.787+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:06.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.787+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:06.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.787+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:06.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.787+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:06.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.787+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:06.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:06.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:06.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:06.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:06.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:06.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:33:06.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:33:06.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:06.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:06.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bed0 con 0x7f63d8005e00 2021-03-09T09:33:06.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:06.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:06.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a910 con 0x7f63d8005e00 2021-03-09T09:33:06.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:33:06.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:06.806 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:06.788+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20002 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:06.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.789+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7353 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc02cf60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:06.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.789+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:33:06.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.789+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:06.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.791+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c870 con 0x7f63d8005e00 2021-03-09T09:33:06.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.791+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:06.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.791+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:06.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.791+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:06.808 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:06.809 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:06.809 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:06.809 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:06.809 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:06.809 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:06.810 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:06.810 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:06.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.791+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03c540 con 0x7f63d8005e00 2021-03-09T09:33:06.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.791+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:06.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.791+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:33:06.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:06.793+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T09:33:07.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:07.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:07.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:07.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:08.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:08.430+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:08.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:08.430+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:08.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:08.430+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:09.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:09.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:09.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:09.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:09.858+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:10.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:10.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:10.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:10.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:11.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:11.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:11.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:11.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:11.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7353 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc02cf60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:11.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7353 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc02cf60).stop 2021-03-09T09:33:11.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:11.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:11.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:11.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:11.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:33:11.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:11.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:11.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:11.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:11.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:11.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:11.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:11.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065e10 con 0x7f63d8003400 2021-03-09T09:33:11.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c900 con 0x7f63d8008840 2021-03-09T09:33:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00678b0 con 0x7f63e0018390 2021-03-09T09:33:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:11.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:11.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:11.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:33:11.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:11.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:11.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:11.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:11.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:11.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:11.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:11.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.789+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:11.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:11.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:11.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:11.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:11.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:11.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:33:11.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:33:11.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:11.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:11.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f5d0 con 0x7f63e0018390 2021-03-09T09:33:11.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:11.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:11.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066760 con 0x7f63e0018390 2021-03-09T09:33:11.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:33:11.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:11.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7364 cs=0 l=1 rev1=1 rx=0x7f63f007ee70 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:11.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:33:11.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.790+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:11.805 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:11.792+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20039 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:11.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007ecb0 con 0x7f63e0018390 2021-03-09T09:33:11.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.792+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:11.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.792+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:11.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.792+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:11.807 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:11.807 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:11.807 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:11.807 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:11.807 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:11.808 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:11.808 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:11.808 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:11.808 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:11.808 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:11.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63e0018390 2021-03-09T09:33:11.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.792+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:11.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.792+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:33:11.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:11.794+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T09:33:12.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:12.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:12.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:12.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:13.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:13.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:13.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:13.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:14.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:14.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:14.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:14.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:14.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:14.859+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:15.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:15.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:15.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:15.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:16.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:16.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:16.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:16.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7364 cs=0 l=1 rev1=1 rx=0x7f63f007ee70 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7364 cs=0 l=1 rev1=1 rx=0x7f63f007ee70 tx=0x7f63f004d180).stop 2021-03-09T09:33:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:16.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:16.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:33:16.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:16.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:16.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:16.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:16.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:16.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:16.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:16.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d670 con 0x7f63d8003400 2021-03-09T09:33:16.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:16.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00678b0 con 0x7f63d8008840 2021-03-09T09:33:16.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:16.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:33:16.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:16.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:16.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:16.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:16.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.792+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:16.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.793+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:16.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.793+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:33:16.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.793+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:16.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.793+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:16.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:16.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:16.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:16.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:16.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:16.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:16.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:16.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:16.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:33:16.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:33:16.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:16.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:16.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016f00 con 0x7f63d8005e00 2021-03-09T09:33:16.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:16.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:16.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:33:16.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:33:16.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.794+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:16.807 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:16.795+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20079 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:16.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.795+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:33:16.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.795+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:16.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.795+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7379 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4009920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:16.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.797+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8005e00 2021-03-09T09:33:16.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.797+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:16.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.797+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:16.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.797+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:16.809 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:16.809 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:16.809 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:16.810 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:16.810 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:16.810 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:16.810 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:16.811 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:16.811 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:16.811 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:16.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.798+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63d8005e00 2021-03-09T09:33:16.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.798+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:16.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.798+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:33:16.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:16.799+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8005e00 2021-03-09T09:33:17.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:17.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:17.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:17.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:18.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:18.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:18.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:18.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:18.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:18.432+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:19.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:19.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:19.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:19.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:19.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:19.860+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:20.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:20.431+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:20.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:20.431+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:21.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:21.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:21.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:21.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:21.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7379 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4009920).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:21.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7379 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4009920).stop 2021-03-09T09:33:21.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:21.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:21.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:21.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:21.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:33:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:21.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:21.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:21.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018100 con 0x7f63d8003400 2021-03-09T09:33:21.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:21.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00678b0 con 0x7f63d8008840 2021-03-09T09:33:21.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:21.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065e10 con 0x7f63e0018390 2021-03-09T09:33:21.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:21.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:21.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:21.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.795+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:21.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.796+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:21.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.796+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:21.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.796+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:33:21.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.796+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:21.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.796+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:21.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.796+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:21.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.796+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:21.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.796+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:21.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:21.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:21.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:21.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:21.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:21.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:33:21.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:33:21.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:21.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:21.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027b10 con 0x7f63e0018390 2021-03-09T09:33:21.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:21.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:21.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f5d0 con 0x7f63e0018390 2021-03-09T09:33:21.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:33:21.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.797+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:21.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.798+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7392 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000eef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:21.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.798+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:33:21.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.798+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:21.811 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:21.798+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20116 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:21.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007bea0 con 0x7f63e0018390 2021-03-09T09:33:21.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.801+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:21.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.801+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:21.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.801+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:21.813 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:21.813 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:21.813 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:21.814 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:21.814 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:21.814 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:21.814 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:21.814 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:21.815 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:21.815 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:21.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0072270 con 0x7f63e0018390 2021-03-09T09:33:21.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.801+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:21.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027760 con 0x7f63e0018390 2021-03-09T09:33:21.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:21.802+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:33:22.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:22.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:22.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:23.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:23.432+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:23.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:23.432+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:24.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:24.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:24.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:24.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:24.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:24.861+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:25.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:25.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:25.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:25.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:26.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:26.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:26.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:26.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:26.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7392 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000eef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:26.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=7392 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000eef0).stop 2021-03-09T09:33:26.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:26.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:26.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:26.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=7392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:33:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T09:33:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:26.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00676b0 con 0x7f63d8008840 2021-03-09T09:33:26.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:26.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c140 con 0x7f63d8005e00 2021-03-09T09:33:26.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:26.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:26.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:33:26.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:26.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.799+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:26.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:26.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:26.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:26.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:26.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:26.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:26.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:26.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:26.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:33:26.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:33:26.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:26.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:26.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b410 con 0x7f63d8005e00 2021-03-09T09:33:26.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:26.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:26.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bed0 con 0x7f63d8005e00 2021-03-09T09:33:26.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:33:26.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.800+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:26.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.801+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7406 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:26.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.801+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:33:26.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.801+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:26.815 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:26.801+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20156 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:26.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.803+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:33:26.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.803+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:26.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.803+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:26.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.803+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:26.817 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:26.817 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:26.817 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:26.818 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:26.818 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:26.818 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:26.818 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:26.819 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:26.819 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:26.819 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:26.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.804+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:33:26.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.804+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:26.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.804+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:33:26.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:26.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04d1d0 con 0x7f63d8005e00 2021-03-09T09:33:27.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:27.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:27.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:27.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:28.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:28.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:28.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:28.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:28.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:28.433+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 msgr2=0x7f63e0025840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T09:33:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 msgr2=0x7f63e0025840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 0x7f63e0025840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f63f0009550 tx=0x7f63f0006b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 0x7f63e0025840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f63f0009550 tx=0x7f63f0006b30).stop 2021-03-09T09:33:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 msgr2=0x7f63e0025840 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T09:33:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 msgr2=0x7f63e0025840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f6400146050 0x7f63e0025840 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f6405d52700 10 monclient: get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:33:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:33:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006b30).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:29.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:29.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:29.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:29.862+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:30.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:30.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:30.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:30.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:31.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:31.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:31.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:31.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7406 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:31.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7406 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc003a50).stop 2021-03-09T09:33:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:33:31.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:31.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:31.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:31.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:33:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:31.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d670 con 0x7f63d8008840 2021-03-09T09:33:31.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:31.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018100 con 0x7f6400146050 2021-03-09T09:33:31.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:31.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:31.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:31.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:31.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:31.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:31.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:33:31.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:31.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:31.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:31.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.802+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:31.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:31.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:33:31.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:33:31.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:31.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:31.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e740 con 0x7f6400146050 2021-03-09T09:33:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b410 con 0x7f6400146050 2021-03-09T09:33:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:33:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.803+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:31.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.804+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 secure :-1 s=READY pgs=7419 cs=0 l=1 rev1=1 rx=0x7f63fc02d070 tx=0x7f63fc021f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:31.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.804+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:33:31.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.804+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:31.818 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:31.804+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20193 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:31.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02bde0 con 0x7f6400146050 2021-03-09T09:33:31.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.805+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:31.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.805+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:31.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.805+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:31.819 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:31.820 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:31.820 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:31.820 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:31.820 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:31.820 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:31.821 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:31.821 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:31.821 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:31.821 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:31.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013c80 con 0x7f6400146050 2021-03-09T09:33:31.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.807+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:31.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.807+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:33:31.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:31.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f6400146050 2021-03-09T09:33:32.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:32.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:32.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:32.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:33.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:33.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:33.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:33.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:34.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:34.433+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:34.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:34.433+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:34.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:34.863+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:35.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:35.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:35.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:35.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:36.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:36.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:36.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 secure :-1 s=READY pgs=7419 cs=0 l=1 rev1=1 rx=0x7f63fc02d070 tx=0x7f63fc021f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 secure :-1 s=READY pgs=7419 cs=0 l=1 rev1=1 rx=0x7f63fc02d070 tx=0x7f63fc021f80).stop 2021-03-09T09:33:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 unknown :-1 s=CLOSED pgs=7419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:36.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:33:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T09:33:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00653b0 con 0x7f63d8008840 2021-03-09T09:33:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029790 con 0x7f63d8005e00 2021-03-09T09:33:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.805+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:36.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.806+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:36.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.806+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:33:36.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.806+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:36.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.806+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:36.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.806+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:36.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.806+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.806+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:36.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:36.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:33:36.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:33:36.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:36.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:36.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a900 con 0x7f63d8005e00 2021-03-09T09:33:36.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027b10 con 0x7f63d8005e00 2021-03-09T09:33:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:33:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7432 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f004ddd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:36.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:33:36.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:36.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:36.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:36.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7432 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f004ddd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:36.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7432 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f004ddd0).stop 2021-03-09T09:33:36.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:36.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:36.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:36.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.807+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:36.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:33:36.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:36.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:36.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:36.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:36.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:36.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:36.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:36.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a220 con 0x7f63d8003400 2021-03-09T09:33:36.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:36.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c3f0 con 0x7f63d8008840 2021-03-09T09:33:36.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:36.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d670 con 0x7f6400146050 2021-03-09T09:33:36.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:36.826 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20233 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:36.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:36.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:36.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:36.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:36.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:36.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:33:36.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:36.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:36.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:36.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:36.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.808+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:36.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:36.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:36.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:33:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:33:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:36.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:36.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002eb10 con 0x7f6400146050 2021-03-09T09:33:36.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:36.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:36.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e740 con 0x7f6400146050 2021-03-09T09:33:36.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:33:36.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.810+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:36.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.811+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 secure :-1 s=READY pgs=7443 cs=0 l=1 rev1=1 rx=0x7f63e000c110 tx=0x7f63fc0712a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:36.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.811+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:33:36.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.811+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:36.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046b40 con 0x7f6400146050 2021-03-09T09:33:36.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.812+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:36.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.812+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:36.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.812+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:36.836 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:36.836 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:36.836 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:36.836 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:36.837 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:36.837 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:36.837 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:36.837 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:36.837 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:36.838 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:36.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046b40 con 0x7f6400146050 2021-03-09T09:33:36.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.813+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:36.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.813+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:33:36.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:36.815+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f300 con 0x7f6400146050 2021-03-09T09:33:37.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:37.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:37.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:38.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:38.434+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:38.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:38.434+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:38.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:38.434+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:39.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:39.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:39.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:39.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:39.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:39.864+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:40.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:40.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:40.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:40.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:41.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:41.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:41.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:33:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 secure :-1 s=READY pgs=7443 cs=0 l=1 rev1=1 rx=0x7f63e000c110 tx=0x7f63fc0712a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 secure :-1 s=READY pgs=7443 cs=0 l=1 rev1=1 rx=0x7f63e000c110 tx=0x7f63fc0712a0).stop 2021-03-09T09:33:41.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025840 unknown :-1 s=CLOSED pgs=7443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:41.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:33:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:41.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029790 con 0x7f63d8003400 2021-03-09T09:33:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8008840 2021-03-09T09:33:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ef0 con 0x7f63d8005e00 2021-03-09T09:33:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:41.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.809+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:41.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.810+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:41.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.810+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:41.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.810+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:41.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.810+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:41.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.810+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:33:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.810+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.810+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:41.821 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20270 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:41.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:41.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:41.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:33:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:33:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f5d0 con 0x7f63d8005e00 2021-03-09T09:33:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002eb10 con 0x7f63d8005e00 2021-03-09T09:33:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:33:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7452 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc05d7e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:33:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.811+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f510 con 0x7f63d8005e00 2021-03-09T09:33:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.812+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.812+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.812+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:41.828 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:41.828 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:41.828 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:41.828 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:41.828 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:41.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032e70 con 0x7f63d8005e00 2021-03-09T09:33:41.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.812+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:41.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.812+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:33:41.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:41.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021ad0 con 0x7f63d8005e00 2021-03-09T09:33:42.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:42.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:42.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:42.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:43.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:43.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:43.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:43.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:44.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:44.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:44.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:44.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:44.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:44.865+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:45.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:45.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:45.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:46.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:46.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:46.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.811+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:46.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.811+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:46.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.811+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7452 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc05d7e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:46.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.811+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7452 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc05d7e0).stop 2021-03-09T09:33:46.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:46.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:46.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:46.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:46.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:33:46.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:46.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:46.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:46.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:46.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0022d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:46.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:46.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:46.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068ce0 con 0x7f63d8003400 2021-03-09T09:33:46.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:46.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8008840 2021-03-09T09:33:46.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:46.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:33:46.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:46.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:46.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:46.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:46.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.812+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.813+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0022d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.813+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:33:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.813+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:46.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.813+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.813+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.813+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.813+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:46.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:33:46.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:33:46.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:46.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:46.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002bda0 con 0x7f6400146050 2021-03-09T09:33:46.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:46.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:46.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a900 con 0x7f6400146050 2021-03-09T09:33:46.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:33:46.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:46.826 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:46.814+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20310 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.815+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0022d50 secure :-1 s=READY pgs=7465 cs=0 l=1 rev1=1 rx=0x7f63f004d660 tx=0x7f63f007eee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.815+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:33:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.815+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007ec70 con 0x7f6400146050 2021-03-09T09:33:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.816+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.816+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.816+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:46.830 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:46.830 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:46.830 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:46.830 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:46.831 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:46.831 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:46.831 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:46.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001ccf0 con 0x7f6400146050 2021-03-09T09:33:46.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.816+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:46.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.816+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:33:46.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:46.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001ccf0 con 0x7f6400146050 2021-03-09T09:33:47.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:47.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:47.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:47.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:48.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:48.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:48.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:48.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:48.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:48.436+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:49.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:49.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:49.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:49.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:49.866+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:50.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:50.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:50.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:50.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:51.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:51.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:51.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0022d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:51.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0022d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:51.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0022d50 secure :-1 s=READY pgs=7465 cs=0 l=1 rev1=1 rx=0x7f63f004d660 tx=0x7f63f007eee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:51.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0022d50 secure :-1 s=READY pgs=7465 cs=0 l=1 rev1=1 rx=0x7f63f004d660 tx=0x7f63f007eee0).stop 2021-03-09T09:33:51.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:51.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:51.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0022d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:51.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0022d50 unknown :-1 s=CLOSED pgs=7465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:51.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:33:51.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:51.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:51.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:51.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:51.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:51.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:51.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:51.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d670 con 0x7f63d8003400 2021-03-09T09:33:51.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:51.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c900 con 0x7f63d8008840 2021-03-09T09:33:51.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:51.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce10 con 0x7f63d8005e00 2021-03-09T09:33:51.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:51.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:51.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:51.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:51.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.815+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:51.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.816+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:51.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.816+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:33:51.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.816+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:51.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.816+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:51.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.817+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:51.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.817+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:51.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.817+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:51.826 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:51.817+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20347 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:51.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:51.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:51.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:51.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:51.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:51.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:33:51.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:33:51.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:51.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:51.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024c20 con 0x7f63d8005e00 2021-03-09T09:33:51.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:51.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:51.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002bda0 con 0x7f63d8005e00 2021-03-09T09:33:51.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:33:51.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:51.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.818+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7482 cs=0 l=1 rev1=1 rx=0x7f63e000c110 tx=0x7f63f007bed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:51.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.819+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:33:51.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.819+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:51.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005d2e0 con 0x7f63d8005e00 2021-03-09T09:33:51.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.819+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:51.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.819+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:51.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.819+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:51.832 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:51.832 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:51.833 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:51.833 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:51.833 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:51.833 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:51.833 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:51.834 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:51.834 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:51.834 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:51.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005d2e0 con 0x7f63d8005e00 2021-03-09T09:33:51.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.823+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:51.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.823+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:33:51.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:51.824+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b80 con 0x7f63d8005e00 2021-03-09T09:33:52.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:52.435+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:52.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:52.435+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:53.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:53.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:53.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:53.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:54.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:54.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:54.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:54.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:54.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:54.867+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:33:55.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:55.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:55.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:55.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:56.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:56.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:56.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:33:56.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:33:56.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7482 cs=0 l=1 rev1=1 rx=0x7f63e000c110 tx=0x7f63f007bed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:33:56.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7482 cs=0 l=1 rev1=1 rx=0x7f63e000c110 tx=0x7f63f007bed0).stop 2021-03-09T09:33:56.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:56.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:33:56.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:33:56.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:33:56.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:33:56.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:56.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:33:56.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:56.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:33:56.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:33:56.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:33:56.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:56.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8003400 2021-03-09T09:33:56.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:56.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:33:56.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:33:56.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ef0 con 0x7f6400146050 2021-03-09T09:33:56.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:33:56.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:33:56.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:56.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:33:56.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.818+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:33:56.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.819+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:33:56.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.819+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:33:56.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.819+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:33:56.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.819+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:33:56.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.819+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:33:56.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.819+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:33:56.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.819+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:33:56.829 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:33:56.820+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20387 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:33:56.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.820+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:33:56.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.820+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:56.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:56.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:33:56.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:33:56.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:33:56.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:33:56.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:56.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:56.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cbb0 con 0x7f6400146050 2021-03-09T09:33:56.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:33:56.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:33:56.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f5d0 con 0x7f6400146050 2021-03-09T09:33:56.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:33:56.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:56.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 secure :-1 s=READY pgs=7493 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:33:56.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:33:56.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.821+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:33:56.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.824+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f510 con 0x7f6400146050 2021-03-09T09:33:56.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.824+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:33:56.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.824+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:33:56.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.824+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:33:56.835 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:33:56.835 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:33:56.835 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:56.836 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:33:56.836 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:33:56.836 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:33:56.836 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:33:56.836 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:33:56.837 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:33:56.837 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:33:56.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.824+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T09:33:56.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.824+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:33:56.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.824+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc033de0 con 0x7f6400146050 2021-03-09T09:33:56.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:56.825+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:33:57.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:57.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:57.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:57.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:58.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:58.437+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:58.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:58.437+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:58.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:58.437+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:33:59.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:59.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:33:59.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:59.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:33:59.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:33:59.868+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:00.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:00.436+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:00.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:00.436+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:01.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:01.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:01.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:01.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:01.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 secure :-1 s=READY pgs=7493 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:01.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 secure :-1 s=READY pgs=7493 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009990).stop 2021-03-09T09:34:01.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:01.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=7493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:34:01.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:01.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:01.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:01.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:01.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:01.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:01.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:01.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T09:34:01.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:01.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069720 con 0x7f63d8008840 2021-03-09T09:34:01.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:01.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d670 con 0x7f63d8005e00 2021-03-09T09:34:01.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:01.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:01.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:01.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:01.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.821+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:01.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.822+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:01.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.822+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:34:01.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.822+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:01.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.822+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.822+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.822+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:01.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.822+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:01.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:01.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:01.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:01.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:01.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:01.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:34:01.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:34:01.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:01.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:01.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a070 con 0x7f63d8005e00 2021-03-09T09:34:01.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:01.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:01.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016f00 con 0x7f63d8005e00 2021-03-09T09:34:01.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:34:01.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:01.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7504 cs=0 l=1 rev1=1 rx=0x7f63f40098b0 tx=0x7f63f4017e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:01.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:34:01.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.823+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:01.837 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:01.824+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20424 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:01.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8005e00 2021-03-09T09:34:01.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.825+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:01.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.825+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:01.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.825+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:01.839 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:01.839 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:01.839 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:01.839 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:01.839 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:01.840 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:01.840 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:01.840 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:01.840 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:01.841 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:01.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.828+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f63d8005e00 2021-03-09T09:34:01.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.828+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:01.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.828+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:34:01.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:01.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8005e00 2021-03-09T09:34:02.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:02.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:02.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:02.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:03.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:03.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:03.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:03.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:04.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:04.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:04.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:04.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:04.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:04.869+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:05.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:05.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:05.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:05.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:06.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:06.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:06.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:06.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:06.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7504 cs=0 l=1 rev1=1 rx=0x7f63f40098b0 tx=0x7f63f4017e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:06.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7504 cs=0 l=1 rev1=1 rx=0x7f63f40098b0 tx=0x7f63f4017e80).stop 2021-03-09T09:34:06.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:06.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:06.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:06.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:06.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:34:06.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:06.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:06.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:06.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:06.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:06.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:06.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:06.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014780 con 0x7f63d8003400 2021-03-09T09:34:06.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:06.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030270 con 0x7f63d8008840 2021-03-09T09:34:06.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:06.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a110 con 0x7f6400146050 2021-03-09T09:34:06.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:06.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:06.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:06.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:06.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:06.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:06.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:34:06.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:06.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:06.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:06.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:06.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.825+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:06.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:06.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:06.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:06.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:06.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:06.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:34:06.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:34:06.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:06.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:06.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00780c0 con 0x7f6400146050 2021-03-09T09:34:06.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:06.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:06.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a070 con 0x7f6400146050 2021-03-09T09:34:06.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:34:06.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.826+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:06.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.827+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=7518 cs=0 l=1 rev1=1 rx=0x7f63f4061ba0 tx=0x7f63f4061a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:06.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.827+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:34:06.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.827+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:06.841 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:06.827+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20464 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:06.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.828+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404be90 con 0x7f6400146050 2021-03-09T09:34:06.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.828+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:06.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.828+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:06.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.828+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:06.842 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:06.843 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:06.843 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:06.843 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:06.843 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:06.843 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:06.844 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:06.844 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:06.844 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:06.844 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:06.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.831+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404be90 con 0x7f6400146050 2021-03-09T09:34:06.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.831+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:06.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.831+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:34:06.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:06.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4034c70 con 0x7f6400146050 2021-03-09T09:34:07.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:07.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:07.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:07.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:08.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:08.438+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:08.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:08.438+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:08.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:08.439+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:09.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:09.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:09.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:09.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:09.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:09.870+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:10.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:10.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:10.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:10.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:11.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:11.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:11.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:34:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=7518 cs=0 l=1 rev1=1 rx=0x7f63f4061ba0 tx=0x7f63f4061a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:11.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=7518 cs=0 l=1 rev1=1 rx=0x7f63f4061ba0 tx=0x7f63f4061a50).stop 2021-03-09T09:34:11.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:11.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:11.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:11.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=CLOSED pgs=7518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:11.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:34:11.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:11.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:11.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:11.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:11.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:11.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:11.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:11.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ef0 con 0x7f63d8003400 2021-03-09T09:34:11.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:11.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:34:11.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:11.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024fd0 con 0x7f63d8005e00 2021-03-09T09:34:11.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:11.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:11.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:11.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:11.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.829+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:11.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:11.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:34:11.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:11.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:11.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:11.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:11.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:11.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:11.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:34:11.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:34:11.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:11.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:11.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024a90 con 0x7f63d8005e00 2021-03-09T09:34:11.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:11.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:11.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cbb0 con 0x7f63d8005e00 2021-03-09T09:34:11.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:34:11.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.830+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:11.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.831+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:34:11.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.831+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:11.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.831+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7530 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a7b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:11.844 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:11.831+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20501 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:11.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00b5d0 con 0x7f63d8005e00 2021-03-09T09:34:11.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.833+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:11.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.833+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:11.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.833+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:11.845 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:11.845 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:11.845 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:11.846 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:11.846 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:11.846 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:11.846 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:11.847 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:11.847 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:11.847 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:11.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.833+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:34:11.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.833+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:11.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.833+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:34:11.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:11.833+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63d8005e00 2021-03-09T09:34:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:12.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:12.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:12.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:13.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:13.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:13.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:13.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:14.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:14.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:14.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:14.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:14.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:14.871+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:15.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:15.439+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:15.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:15.439+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:16.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:16.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:16.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:34:16.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:16.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7530 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a7b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:16.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7530 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a7b0).stop 2021-03-09T09:34:16.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:16.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:16.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:16.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:16.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:34:16.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:16.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:16.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:16.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:16.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:16.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:16.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:16.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069720 con 0x7f63d8003400 2021-03-09T09:34:16.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:16.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b2f0 con 0x7f63d8008840 2021-03-09T09:34:16.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:16.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024fd0 con 0x7f6400146050 2021-03-09T09:34:16.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:16.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:16.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.832+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:16.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.833+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:16.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.833+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:16.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.833+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:16.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.833+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:34:16.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.833+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:16.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.833+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.833+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:16.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.833+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:16.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.833+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:16.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:16.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:16.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:16.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:16.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:16.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:34:16.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:34:16.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:16.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:16.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f6f0 con 0x7f6400146050 2021-03-09T09:34:16.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:16.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:16.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024c20 con 0x7f6400146050 2021-03-09T09:34:16.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:34:16.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:16.857 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:16.834+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20541 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.835+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=7545 cs=0 l=1 rev1=1 rx=0x7f63f001cc80 tx=0x7f63f0088fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.835+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:34:16.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.835+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f6400146050 2021-03-09T09:34:16.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.836+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.836+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:16.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.836+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:16.862 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:16.862 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:16.862 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:16.863 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:16.863 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:16.864 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:16.864 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:16.864 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:16.865 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:16.865 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f6400146050 2021-03-09T09:34:16.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.839+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.839+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:34:16.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:16.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b7c0 con 0x7f6400146050 2021-03-09T09:34:17.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:17.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:17.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:17.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:18.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:18.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:18.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:18.440+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:19.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:19.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:19.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:19.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:19.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:19.872+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:20.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:20.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:20.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:20.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:21.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:21.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:21.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:21.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:21.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=7545 cs=0 l=1 rev1=1 rx=0x7f63f001cc80 tx=0x7f63f0088fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:21.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 secure :-1 s=READY pgs=7545 cs=0 l=1 rev1=1 rx=0x7f63f001cc80 tx=0x7f63f0088fe0).stop 2021-03-09T09:34:21.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:21.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:21.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:21.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002da40 unknown :-1 s=CLOSED pgs=7545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:21.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:34:21.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:21.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:21.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:21.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:21.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:21.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:21.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:21.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ef0 con 0x7f63d8003400 2021-03-09T09:34:21.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:21.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:34:21.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:21.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066010 con 0x7f63d8005e00 2021-03-09T09:34:21.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:21.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:21.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:21.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:34:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:21.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:21.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.835+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:21.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:21.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:21.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:21.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:34:21.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:34:21.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:21.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:21.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024c20 con 0x7f63d8005e00 2021-03-09T09:34:21.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:21.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:21.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024a90 con 0x7f63d8005e00 2021-03-09T09:34:21.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:34:21.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.836+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:21.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.837+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:34:21.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.837+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:21.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.837+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7557 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc02b420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:21.849 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:21.838+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20578 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f63d8005e00 2021-03-09T09:34:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.840+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.840+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:21.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.840+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:21.851 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:21.852 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:21.852 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:21.852 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:21.852 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:21.853 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02b810 con 0x7f63d8005e00 2021-03-09T09:34:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.840+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8005e00 2021-03-09T09:34:21.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:21.840+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:34:22.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:22.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:22.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:22.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:23.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:23.440+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:23.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:23.440+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:24.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:24.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:24.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:24.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:24.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:24.873+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:25.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:25.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:25.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:25.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:26.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:26.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:26.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:34:26.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:26.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7557 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc02b420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:26.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7557 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc02b420).stop 2021-03-09T09:34:26.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:26.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:26.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:26.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:26.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:34:26.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:26.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:26.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:26.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:26.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:26.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:26.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:26.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d860 con 0x7f63d8003400 2021-03-09T09:34:26.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:26.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d670 con 0x7f63d8008840 2021-03-09T09:34:26.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:26.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b2f0 con 0x7f6400146050 2021-03-09T09:34:26.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:26.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:26.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.838+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:26.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.839+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:26.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.839+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:26.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.839+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:26.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.839+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:34:26.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.839+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:26.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.839+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:26.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.839+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:26.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.839+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:26.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.839+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:26.855 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20618 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:26.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:26.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:26.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:26.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:26.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:26.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:34:26.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:34:26.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:26.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:26.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019c80 con 0x7f6400146050 2021-03-09T09:34:26.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:26.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:26.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f6f0 con 0x7f6400146050 2021-03-09T09:34:26.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:34:26.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:26.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ec60 secure :-1 s=READY pgs=7578 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f007bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:26.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:34:26.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.841+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:26.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001cec0 con 0x7f6400146050 2021-03-09T09:34:26.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.842+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:26.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.842+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:26.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.842+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:26.867 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:26.868 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:26.868 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:26.869 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:26.869 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:26.869 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:26.870 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:26.870 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:26.871 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:26.871 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:26.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f6400146050 2021-03-09T09:34:26.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.844+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:26.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.844+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:34:26.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:26.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:34:27.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:27.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:27.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:27.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:28.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:28.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:28.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:28.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:28.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:28.441+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:29.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:29.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:29.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:29.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:29.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:29.874+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:30.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:30.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:30.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:30.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:31.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:31.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:31.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:34:31.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:31.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ec60 secure :-1 s=READY pgs=7578 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f007bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:31.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ec60 secure :-1 s=READY pgs=7578 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f007bf40).stop 2021-03-09T09:34:31.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:31.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:31.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002ec60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:31.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002ec60 unknown :-1 s=CLOSED pgs=7578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:31.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:34:31.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:31.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:31.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:31.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:31.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:31.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:31.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:31.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ec50 con 0x7f63d8003400 2021-03-09T09:34:31.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:31.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8008840 2021-03-09T09:34:31.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:31.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b000 con 0x7f63d8005e00 2021-03-09T09:34:31.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:31.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:31.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.841+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:31.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:31.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.842+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:31.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:31.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.842+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:34:31.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.842+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:31.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.842+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:31.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.842+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:31.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.842+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:31.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.842+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:31.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:31.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:31.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:31.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:31.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:31.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:34:31.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:34:31.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:31.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:31.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00230b0 con 0x7f63d8005e00 2021-03-09T09:34:31.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:31.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:31.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019c80 con 0x7f63d8005e00 2021-03-09T09:34:31.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:34:31.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.843+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:31.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.844+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:34:31.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.844+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:31.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7591 cs=0 l=1 rev1=1 rx=0x7f63f0027de0 tx=0x7f63f000ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:31.862 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:31.844+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20655 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:31.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:34:31.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.845+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:31.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.845+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:31.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.845+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:31.864 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:31.864 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:31.864 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:31.864 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:31.864 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:31.865 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:31.865 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:31.865 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:31.865 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:31.866 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:31.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:34:31.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.846+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:31.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.846+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:34:31.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:31.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036a90 con 0x7f63d8005e00 2021-03-09T09:34:32.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:32.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:32.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:32.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:33.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:33.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:33.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:33.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:34.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:34.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:34.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:34.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:34.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:34.875+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:35.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:35.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:35.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:35.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:36.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:36.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:36.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:34:36.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:36.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7591 cs=0 l=1 rev1=1 rx=0x7f63f0027de0 tx=0x7f63f000ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:36.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7591 cs=0 l=1 rev1=1 rx=0x7f63f0027de0 tx=0x7f63f000ff20).stop 2021-03-09T09:34:36.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:36.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:36.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:36.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:36.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:34:36.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:36.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:36.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:36.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:36.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:36.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:36.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:36.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066010 con 0x7f63d8003400 2021-03-09T09:34:36.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:36.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ef0 con 0x7f63d8008840 2021-03-09T09:34:36.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:36.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:34:36.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:36.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:36.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:36.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:36.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:36.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:36.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:34:36.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:36.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.844+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:36.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:36.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:36.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.845+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:36.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:36.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:36.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:36.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:36.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:36.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:34:36.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:34:36.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:36.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:36.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d810 con 0x7f6400146050 2021-03-09T09:34:36.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:36.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:36.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00780c0 con 0x7f6400146050 2021-03-09T09:34:36.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:34:36.858 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20697 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:36.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.846+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:36.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 secure :-1 s=READY pgs=7606 cs=0 l=1 rev1=1 rx=0x7f63f4061ac0 tx=0x7f63f4003c40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:36.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.847+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:34:36.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.847+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:36.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:34:36.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.848+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:36.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.848+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:36.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.848+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:36.862 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:36.862 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:36.862 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:36.862 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:36.863 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:36.863 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:36.863 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:36.863 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:36.863 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:36.864 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:36.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:34:36.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.852+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:36.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.852+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:34:36.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:36.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f6400146050 2021-03-09T09:34:37.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:37.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:37.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:37.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:38.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:38.441+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:38.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:38.441+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:38.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:38.441+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:39.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:39.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:39.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:39.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:39.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:39.875+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:40.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:40.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:40.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:40.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:41.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:41.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:41.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:34:41.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:41.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 secure :-1 s=READY pgs=7606 cs=0 l=1 rev1=1 rx=0x7f63f4061ac0 tx=0x7f63f4003c40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:41.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 secure :-1 s=READY pgs=7606 cs=0 l=1 rev1=1 rx=0x7f63f4061ac0 tx=0x7f63f4003c40).stop 2021-03-09T09:34:41.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:41.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:41.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:41.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 unknown :-1 s=CLOSED pgs=7606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:41.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:34:41.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:41.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:41.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:41.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:41.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:41.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:41.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:41.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069720 con 0x7f63d8003400 2021-03-09T09:34:41.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:41.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b2f0 con 0x7f63d8008840 2021-03-09T09:34:41.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:41.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T09:34:41.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:41.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:41.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:41.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:41.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:41.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.848+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:41.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.848+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:34:41.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.848+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:41.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.848+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:41.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.848+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:41.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.848+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:41.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:41.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:41.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:41.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:41.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:41.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:41.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:34:41.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:34:41.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:41.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:41.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024a90 con 0x7f63d8005e00 2021-03-09T09:34:41.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:41.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:41.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024c20 con 0x7f63d8005e00 2021-03-09T09:34:41.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:34:41.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.849+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:41.863 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:41.850+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20732 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:41.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.850+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:34:41.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d500 secure :-1 s=READY pgs=7617 cs=0 l=1 rev1=1 rx=0x7f63fc051bf0 tx=0x7f63fc06f4a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:41.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.850+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:41.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:34:41.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.851+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:41.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.851+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:41.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.851+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:41.865 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:41.865 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:41.866 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:41.866 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:41.866 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:41.866 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:41.866 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:41.867 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:41.867 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:41.867 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:41.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051c20 con 0x7f63d8005e00 2021-03-09T09:34:41.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.851+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:41.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.851+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:34:41.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:41.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:34:42.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:42.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:42.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:42.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:43.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:43.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:43.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:43.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:44.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:44.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:44.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:44.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:44.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:44.876+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:45.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:45.442+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:45.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:45.442+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:46.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:46.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:46.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d500 secure :-1 s=READY pgs=7617 cs=0 l=1 rev1=1 rx=0x7f63fc051bf0 tx=0x7f63fc06f4a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:46.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d500 secure :-1 s=READY pgs=7617 cs=0 l=1 rev1=1 rx=0x7f63fc051bf0 tx=0x7f63fc06f4a0).stop 2021-03-09T09:34:46.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:46.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:46.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:46.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d500 unknown :-1 s=CLOSED pgs=7617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:46.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:34:46.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:46.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:46.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:46.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:46.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:46.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:46.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:46.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ee50 con 0x7f63d8003400 2021-03-09T09:34:46.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:46.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:34:46.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:46.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ef0 con 0x7f6400146050 2021-03-09T09:34:46.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.850+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:46.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.851+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:46.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:46.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:46.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:46.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:46.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.851+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:34:46.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.851+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:46.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.851+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:46.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.852+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:46.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.852+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:46.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.852+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:46.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:46.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:46.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:46.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:46.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:46.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:34:46.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:34:46.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:46.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:46.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b320 con 0x7f6400146050 2021-03-09T09:34:46.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:46.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:46.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d810 con 0x7f6400146050 2021-03-09T09:34:46.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:34:46.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:46.865 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:46.853+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20773 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:46.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.854+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:34:46.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.854+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:46.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.854+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bf80 secure :-1 s=READY pgs=7632 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4072990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:46.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072d20 con 0x7f6400146050 2021-03-09T09:34:46.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.857+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:46.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.857+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:46.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.857+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:46.868 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:46.868 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:46.868 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:46.868 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:46.869 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:46.869 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:46.869 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:46.869 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:46.869 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:46.870 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:46.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:34:46.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.860+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:46.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.860+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:34:46.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:46.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4072d20 con 0x7f6400146050 2021-03-09T09:34:47.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:47.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:47.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:47.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:48.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:48.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:48.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:48.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:48.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:48.443+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:49.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:49.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:49.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:49.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:49.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:49.877+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:50.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:50.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:50.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:50.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:51.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:51.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:51.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:34:51.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.853+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:51.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.853+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bf80 secure :-1 s=READY pgs=7632 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4072990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:51.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.853+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bf80 secure :-1 s=READY pgs=7632 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4072990).stop 2021-03-09T09:34:51.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.853+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:51.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.853+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:51.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:51.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bf80 unknown :-1 s=CLOSED pgs=7632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:51.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:34:51.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:51.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:51.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:51.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:51.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:51.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:51.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:51.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b000 con 0x7f63d8003400 2021-03-09T09:34:51.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:51.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011770 con 0x7f63d8008840 2021-03-09T09:34:51.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:51.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015400 con 0x7f63d8005e00 2021-03-09T09:34:51.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:51.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:51.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:51.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:51.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:51.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:51.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:34:51.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:51.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.854+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:51.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:51.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:51.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:51.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:51.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:51.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:51.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:51.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:51.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:34:51.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:34:51.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:51.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:51.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bea0 con 0x7f63d8005e00 2021-03-09T09:34:51.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:51.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:51.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024a90 con 0x7f63d8005e00 2021-03-09T09:34:51.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:34:51.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.855+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:51.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.856+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:34:51.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.856+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7643 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc04d610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:51.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.856+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:51.869 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:51.856+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20811 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:51.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63d8005e00 2021-03-09T09:34:51.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.858+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:51.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.858+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:51.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.858+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:51.871 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:51.871 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:51.872 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:51.872 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:51.872 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:51.872 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:51.873 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:51.873 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:51.873 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:51.873 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:51.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:34:51.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.858+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:51.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.859+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:34:51.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:51.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:34:52.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:52.443+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:52.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:52.443+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:53.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:53.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:53.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:53.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:54.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:54.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:54.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:54.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:54.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:54.878+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:34:55.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:55.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:55.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:55.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:56.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:56.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:56.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:34:56.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:34:56.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7643 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc04d610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:34:56.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7643 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc04d610).stop 2021-03-09T09:34:56.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:56.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:34:56.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:34:56.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:34:56.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:34:56.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:56.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:34:56.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:56.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:34:56.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:34:56.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:34:56.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:56.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069720 con 0x7f63d8003400 2021-03-09T09:34:56.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:56.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068410 con 0x7f63d8008840 2021-03-09T09:34:56.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:34:56.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018220 con 0x7f6400146050 2021-03-09T09:34:56.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:34:56.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:34:56.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:56.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.857+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:34:56.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:34:56.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.858+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:34:56.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.858+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:34:56.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.858+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:34:56.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.858+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:34:56.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:34:56.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:34:56.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:34:56.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:34:56.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:56.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:56.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:34:56.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:34:56.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:34:56.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:34:56.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:56.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:56.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d810 con 0x7f6400146050 2021-03-09T09:34:56.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:34:56.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:34:56.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00230b0 con 0x7f6400146050 2021-03-09T09:34:56.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:34:56.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.859+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:56.874 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:34:56.860+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20850 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:34:56.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.860+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:34:56.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c690 secure :-1 s=READY pgs=7660 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f0006680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:34:56.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.860+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:34:56.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f6400146050 2021-03-09T09:34:56.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.861+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:34:56.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.861+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:34:56.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.861+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:34:56.879 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:34:56.879 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:34:56.880 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:56.880 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:34:56.881 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:34:56.881 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:34:56.881 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:34:56.882 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:34:56.882 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:34:56.883 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:34:56.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f6400146050 2021-03-09T09:34:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.861+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:34:56.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.861+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:34:56.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:56.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:34:57.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:57.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:57.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:57.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:58.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:58.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:58.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:58.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:58.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:58.444+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:34:59.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:59.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:34:59.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:59.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:34:59.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:34:59.879+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:00.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:00.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:00.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:00.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:01.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.444+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:01.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.444+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:01.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:35:01.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:01.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c690 secure :-1 s=READY pgs=7660 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f0006680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:01.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c690 secure :-1 s=READY pgs=7660 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f0006680).stop 2021-03-09T09:35:01.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:01.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:01.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:01.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c690 unknown :-1 s=CLOSED pgs=7660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:01.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:35:01.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:01.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:01.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:01.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:01.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:01.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:01.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:01.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022140 con 0x7f63d8003400 2021-03-09T09:35:01.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:01.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011770 con 0x7f63d8008840 2021-03-09T09:35:01.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:01.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:35:01.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.860+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:01.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.861+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:01.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.861+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:01.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.861+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:01.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.861+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:01.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.861+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:01.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.861+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:35:01.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.861+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:01.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.861+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:01.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.862+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:01.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.862+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:01.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.862+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:01.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:01.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:01.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:01.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:01.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:01.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:35:01.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:35:01.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:01.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:01.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ea30 con 0x7f63d8005e00 2021-03-09T09:35:01.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:01.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:01.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b320 con 0x7f63d8005e00 2021-03-09T09:35:01.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:35:01.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:01.885 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:01.863+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20888 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:01.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.864+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7676 cs=0 l=1 rev1=1 rx=0x7f63f4009c20 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:01.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.864+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:35:01.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.864+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:01.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63d8005e00 2021-03-09T09:35:01.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.865+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:01.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.865+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:01.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.865+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:01.890 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:01.890 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:01.891 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:01.891 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:01.891 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:01.892 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:01.892 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:01.893 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:01.893 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:01.893 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:01.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63d8005e00 2021-03-09T09:35:01.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.867+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:01.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.867+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:35:01.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:01.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400e770 con 0x7f63d8005e00 2021-03-09T09:35:02.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:02.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:02.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:02.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:03.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:03.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:03.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:03.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:04.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:04.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:04.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:04.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:04.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:04.880+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:05.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:05.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:05.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:05.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:06.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:06.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.863+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:06.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.863+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:06.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.863+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7676 cs=0 l=1 rev1=1 rx=0x7f63f4009c20 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:06.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.863+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7676 cs=0 l=1 rev1=1 rx=0x7f63f4009c20 tx=0x7f63f4017ed0).stop 2021-03-09T09:35:06.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.863+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:06.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.863+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:06.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.863+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:35:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:06.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:06.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:06.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:06.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:06.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024e70 con 0x7f63d8003400 2021-03-09T09:35:06.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:06.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069720 con 0x7f63d8008840 2021-03-09T09:35:06.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:06.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f6400146050 2021-03-09T09:35:06.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:06.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:06.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:06.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:06.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.864+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:06.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.865+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:06.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.865+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:35:06.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.865+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:06.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.865+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:06.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.866+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:06.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.866+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:06.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.866+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:06.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.866+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:06.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:06.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:06.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:06.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:06.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:35:06.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:35:06.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:06.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:06.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f6400146050 2021-03-09T09:35:06.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:06.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:06.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ea30 con 0x7f6400146050 2021-03-09T09:35:06.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:35:06.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.867+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:06.883 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:06.866+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20926 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:06.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.868+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:35:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.868+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 secure :-1 s=READY pgs=7690 cs=0 l=1 rev1=1 rx=0x7f63f4034bf0 tx=0x7f63f40051f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400e260 con 0x7f6400146050 2021-03-09T09:35:06.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.871+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:06.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.871+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:06.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.871+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:06.885 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:06.885 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:06.885 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:06.886 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:06.886 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:06.886 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:06.886 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:06.887 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:06.887 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:06.887 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:06.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400e3c0 con 0x7f6400146050 2021-03-09T09:35:06.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.871+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:06.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.871+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:35:06.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:06.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f6400146050 2021-03-09T09:35:07.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:07.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:07.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:07.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:08.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:08.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:08.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:08.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:08.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:08.445+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:09.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:09.445+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:09.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:09.445+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:09.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:09.881+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:10.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:10.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:10.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:11.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:11.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 secure :-1 s=READY pgs=7690 cs=0 l=1 rev1=1 rx=0x7f63f4034bf0 tx=0x7f63f40051f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 secure :-1 s=READY pgs=7690 cs=0 l=1 rev1=1 rx=0x7f63f4034bf0 tx=0x7f63f40051f0).stop 2021-03-09T09:35:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 unknown :-1 s=CLOSED pgs=7690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:35:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:11.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8003400 2021-03-09T09:35:11.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:11.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022b40 con 0x7f63d8008840 2021-03-09T09:35:11.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:35:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.867+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.868+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.868+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.868+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.868+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:35:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.868+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.868+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.868+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.868+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.868+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:35:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:35:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:35:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bea0 con 0x7f63d8005e00 2021-03-09T09:35:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:35:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.869+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:11.883 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:11.870+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 20963 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.870+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7700 cs=0 l=1 rev1=1 rx=0x7f63e0022480 tx=0x7f63fc051f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.870+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:35:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.870+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63d8005e00 2021-03-09T09:35:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.874+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.874+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.874+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:11.886 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:11.886 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:11.886 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:11.886 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:11.887 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:11.887 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:11.887 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:11.887 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:11.887 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:11.888 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:11.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:35:11.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.874+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:11.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.874+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:35:11.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:11.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:35:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:12.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:12.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:12.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:13.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:13.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:13.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:13.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:14.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:14.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:14.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:14.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:14.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:14.882+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:15.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:15.446+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:15.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:15.446+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:16.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:16.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:16.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:16.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7700 cs=0 l=1 rev1=1 rx=0x7f63e0022480 tx=0x7f63fc051f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7700 cs=0 l=1 rev1=1 rx=0x7f63e0022480 tx=0x7f63fc051f40).stop 2021-03-09T09:35:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:16.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:35:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:16.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0024050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:16.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:16.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:16.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022140 con 0x7f63d8003400 2021-03-09T09:35:16.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:16.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011770 con 0x7f63d8008840 2021-03-09T09:35:16.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:16.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f040 con 0x7f6400146050 2021-03-09T09:35:16.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.870+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:16.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.871+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:16.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.871+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:16.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0024050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:16.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.871+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:35:16.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.871+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:16.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.871+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:16.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.871+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:16.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.871+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:16.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:16.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:16.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:16.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:16.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:16.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:16.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:35:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:35:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00230b0 con 0x7f6400146050 2021-03-09T09:35:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d810 con 0x7f6400146050 2021-03-09T09:35:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:35:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0024050 secure :-1 s=READY pgs=7714 cs=0 l=1 rev1=1 rx=0x7f63f001cff0 tx=0x7f63f000ef60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:16.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:35:16.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:16.891 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:16.873+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21003 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:16.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:35:16.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.876+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:16.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.876+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:16.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.876+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:16.893 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:16.893 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:16.893 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:16.894 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:16.895 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:35:16.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.878+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.878+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:35:16.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:16.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005eb10 con 0x7f6400146050 2021-03-09T09:35:17.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:17.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:17.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:17.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:18.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:18.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:18.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:18.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:18.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:18.447+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:19.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:19.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:19.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:19.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:19.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:19.883+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:20.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:20.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:20.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:20.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:21.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:21.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:21.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0024050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:21.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0024050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:21.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0024050 secure :-1 s=READY pgs=7714 cs=0 l=1 rev1=1 rx=0x7f63f001cff0 tx=0x7f63f000ef60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:21.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0024050 secure :-1 s=READY pgs=7714 cs=0 l=1 rev1=1 rx=0x7f63f001cff0 tx=0x7f63f000ef60).stop 2021-03-09T09:35:21.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:21.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:21.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0024050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:21.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0024050 unknown :-1 s=CLOSED pgs=7714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:21.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:35:21.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:21.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:21.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:21.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:21.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:21.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:21.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:21.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069720 con 0x7f63d8003400 2021-03-09T09:35:21.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:21.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065090 con 0x7f63d8008840 2021-03-09T09:35:21.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:21.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8005e00 2021-03-09T09:35:21.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:21.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:21.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:21.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:21.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:21.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:21.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:35:21.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:21.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:21.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:21.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:21.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:21.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:21.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:21.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:21.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:35:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:35:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:21.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022e80 con 0x7f63d8005e00 2021-03-09T09:35:21.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:21.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:21.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:35:21.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:35:21.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.875+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:21.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.876+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7722 cs=0 l=1 rev1=1 rx=0x7f63e006d3d0 tx=0x7f63fc052e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:21.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.876+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:35:21.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.876+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:21.890 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:21.877+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21042 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:21.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:35:21.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.877+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:21.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.877+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.877+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:21.893 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:21.893 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:21.893 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:21.893 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:21.893 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:21.894 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:21.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc066e90 con 0x7f63d8005e00 2021-03-09T09:35:21.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.878+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:21.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.878+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:35:21.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:21.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:35:22.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:22.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:22.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:22.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:23.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:23.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:23.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:23.447+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:24.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:24.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:24.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:24.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:24.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:24.884+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:25.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:25.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:25.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:25.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:26.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:26.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:26.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:26.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:26.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7722 cs=0 l=1 rev1=1 rx=0x7f63e006d3d0 tx=0x7f63fc052e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:26.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7722 cs=0 l=1 rev1=1 rx=0x7f63e006d3d0 tx=0x7f63fc052e80).stop 2021-03-09T09:35:26.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:26.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:26.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:26.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:26.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:35:26.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:26.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:26.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:26.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:26.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:26.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:26.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:26.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023cb0 con 0x7f63d8003400 2021-03-09T09:35:26.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:26.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011770 con 0x7f63d8008840 2021-03-09T09:35:26.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:26.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3c0 con 0x7f6400146050 2021-03-09T09:35:26.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.877+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:26.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:26.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.878+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:26.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.878+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:26.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.878+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:26.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:26.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:35:26.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:26.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:26.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:26.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:26.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:26.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:26.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:26.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:26.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:26.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:26.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:35:26.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:35:26.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:26.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:26.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bea0 con 0x7f6400146050 2021-03-09T09:35:26.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:26.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:26.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f6400146050 2021-03-09T09:35:26.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:35:26.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.879+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:26.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.880+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017530 secure :-1 s=READY pgs=7740 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f4063250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:26.893 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:26.880+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21078 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:26.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.880+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:35:26.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.880+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:26.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f6400146050 2021-03-09T09:35:26.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.884+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:26.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.884+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:26.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.884+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:26.895 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:26.895 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:26.895 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:26.895 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:26.897 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:26.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f6400146050 2021-03-09T09:35:26.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.884+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:26.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005b70 con 0x7f6400146050 2021-03-09T09:35:26.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:26.884+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:35:27.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:27.447+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:27.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:27.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:28.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:28.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:28.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:28.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:28.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:28.448+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:29.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:29.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:29.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:29.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:29.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:29.885+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:30.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:30.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:30.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:30.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:31.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:31.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:31.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:31.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:31.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017530 secure :-1 s=READY pgs=7740 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f4063250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:31.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017530 secure :-1 s=READY pgs=7740 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f4063250).stop 2021-03-09T09:35:31.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:31.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:31.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0017530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:31.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0017530 unknown :-1 s=CLOSED pgs=7740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:31.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:35:31.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:31.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:31.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:31.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:31.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:31.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:31.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:31.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:35:31.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:31.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d320 con 0x7f63d8008840 2021-03-09T09:35:31.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:31.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023cb0 con 0x7f63d8005e00 2021-03-09T09:35:31.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:31.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:31.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:31.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:31.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.881+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:31.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.882+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:31.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.882+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:35:31.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.882+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:31.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.882+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:31.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.882+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:31.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.882+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:31.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.882+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:31.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.882+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:31.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:31.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:31.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:31.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:31.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:35:31.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:35:31.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:31.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:31.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d810 con 0x7f63d8005e00 2021-03-09T09:35:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00230b0 con 0x7f63d8005e00 2021-03-09T09:35:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:35:31.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:31.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:35:31.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:31.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.883+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7753 cs=0 l=1 rev1=1 rx=0x7f63e003c420 tx=0x7f63f004dab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:31.897 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:31.884+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21119 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:31.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:35:31.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.885+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:31.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.885+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:31.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.885+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:31.899 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:31.899 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:31.899 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:31.900 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:31.900 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:31.900 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:31.900 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:31.901 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:31.901 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:31.901 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:31.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:35:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.885+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.885+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:35:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:31.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T09:35:32.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:32.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:32.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:32.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:33.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:33.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:33.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:33.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:34.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:34.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:34.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:34.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:34.886+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:35.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:35.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:35.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:35.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:36.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:36.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:36.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:36.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:36.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7753 cs=0 l=1 rev1=1 rx=0x7f63e003c420 tx=0x7f63f004dab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:36.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7753 cs=0 l=1 rev1=1 rx=0x7f63e003c420 tx=0x7f63f004dab0).stop 2021-03-09T09:35:36.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:36.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:36.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:36.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:36.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:35:36.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:36.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:36.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:36.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:36.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:36.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:36.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:36.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006acd0 con 0x7f63d8003400 2021-03-09T09:35:36.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:36.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b4c0 con 0x7f63d8008840 2021-03-09T09:35:36.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:36.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016f50 con 0x7f6400146050 2021-03-09T09:35:36.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.884+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:36.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:36.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:36.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:36.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:36.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:36.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:35:36.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:36.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:36.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:36.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:36.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:36.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.885+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:36.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:36.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:36.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:36.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:36.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:35:36.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:35:36.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:36.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:36.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ab90 con 0x7f6400146050 2021-03-09T09:35:36.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:36.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:36.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022e80 con 0x7f6400146050 2021-03-09T09:35:36.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:35:36.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:36.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 secure :-1 s=READY pgs=7765 cs=0 l=1 rev1=1 rx=0x7f63e0011ab0 tx=0x7f63fc00b640).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:36.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:35:36.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.886+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:36.901 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:36.887+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21157 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:36.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052770 con 0x7f6400146050 2021-03-09T09:35:36.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.888+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:36.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.888+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:36.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.888+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:36.903 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:36.903 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:36.903 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:36.904 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:36.904 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:36.904 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:36.904 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:36.905 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:36.905 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:36.905 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:36.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:35:36.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.892+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:36.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.892+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:35:36.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:36.893+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc052770 con 0x7f6400146050 2021-03-09T09:35:37.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:37.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:37.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:37.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:38.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:38.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:38.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:38.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:38.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:38.449+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:39.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:39.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:39.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:39.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:39.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:39.887+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:40.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:40.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:40.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:40.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:41.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:35:41.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:41.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 secure :-1 s=READY pgs=7765 cs=0 l=1 rev1=1 rx=0x7f63e0011ab0 tx=0x7f63fc00b640).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:41.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 secure :-1 s=READY pgs=7765 cs=0 l=1 rev1=1 rx=0x7f63e0011ab0 tx=0x7f63fc00b640).stop 2021-03-09T09:35:41.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:41.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:41.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:41.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f290 unknown :-1 s=CLOSED pgs=7765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:41.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:35:41.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:41.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:41.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:41.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:41.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:41.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:41.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:41.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ef0 con 0x7f63d8003400 2021-03-09T09:35:41.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:41.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8008840 2021-03-09T09:35:41.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:41.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T09:35:41.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:41.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:41.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:41.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:41.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.888+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:41.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.889+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:41.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.889+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:35:41.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.889+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:41.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.889+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:41.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.889+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:41.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.889+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:41.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.889+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:41.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:41.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:41.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:41.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:41.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:41.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:35:41.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:35:41.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:41.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:41.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016370 con 0x7f63d8005e00 2021-03-09T09:35:41.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:41.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:41.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bea0 con 0x7f63d8005e00 2021-03-09T09:35:41.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:35:41.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:41.905 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:41.890+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21196 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:41.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.891+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:35:41.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.891+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:41.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.891+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7785 cs=0 l=1 rev1=1 rx=0x7f63f406aa00 tx=0x7f63f404d290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:41.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404d450 con 0x7f63d8005e00 2021-03-09T09:35:41.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.892+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:41.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.892+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:41.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.892+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:41.907 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:41.907 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:41.908 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:41.908 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:41.908 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:41.908 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:41.909 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:41.909 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:41.909 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:41.909 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:41.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404d450 con 0x7f63d8005e00 2021-03-09T09:35:41.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.894+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:41.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.894+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:35:41.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:41.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63d8005e00 2021-03-09T09:35:42.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:42.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:42.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:42.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:43.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:43.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:43.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:43.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:44.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:44.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:44.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:44.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:44.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:44.888+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:45.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:45.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:45.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:45.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:46.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:46.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:46.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:46.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:46.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7785 cs=0 l=1 rev1=1 rx=0x7f63f406aa00 tx=0x7f63f404d290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:46.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7785 cs=0 l=1 rev1=1 rx=0x7f63f406aa00 tx=0x7f63f404d290).stop 2021-03-09T09:35:46.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:46.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:46.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:46.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:46.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:35:46.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:46.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:46.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:46.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019250 con 0x7f63d8003400 2021-03-09T09:35:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:35:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:46.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d120 con 0x7f6400146050 2021-03-09T09:35:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:46.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:46.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.891+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:46.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.892+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.892+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:35:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.892+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.892+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.893+0000 7f6405d52700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T09:35:46.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.893+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.893+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.893+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.893+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2021-03-09T09:35:46.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.893+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).write auth request more write failed r=-32 ((32) Broken pipe) 2021-03-09T09:35:46.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:46.893+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:46.903 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:46.894+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21234 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:47.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.092+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:47.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.092+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:47.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.092+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T09:35:47.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.092+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T09:35:47.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:47.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:35:47.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:47.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:47.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:47.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:47.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:47.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:47.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:47.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:47.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:47.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:47.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:35:47.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:35:47.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:47.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:47.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005dbd0 con 0x7f6400146050 2021-03-09T09:35:47.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:47.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:47.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d810 con 0x7f6400146050 2021-03-09T09:35:47.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:35:47.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:47.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:35:47.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:47.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 secure :-1 s=READY pgs=7808 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:47.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00541f0 con 0x7f6400146050 2021-03-09T09:35:47.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.095+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:47.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.095+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:47.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.095+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:47.105 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:47.105 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:47.106 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:47.106 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:47.106 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:47.106 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:47.107 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:47.107 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:47.107 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:47.107 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:47.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:35:47.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.095+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:47.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.095+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:35:47.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.096+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036bc0 con 0x7f6400146050 2021-03-09T09:35:47.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:47.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:47.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:48.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:48.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:48.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:48.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:48.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:48.451+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:49.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:49.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:49.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:49.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:49.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:49.889+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:50.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:50.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:50.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:50.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:51.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:51.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:51.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:51.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 secure :-1 s=READY pgs=7808 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:51.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 secure :-1 s=READY pgs=7808 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c40).stop 2021-03-09T09:35:51.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:51.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:51.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:51.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 unknown :-1 s=CLOSED pgs=7808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:51.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:35:51.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:51.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:51.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:51.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.895+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:51.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:51.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:51.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:51.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f7b0 con 0x7f63d8003400 2021-03-09T09:35:51.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:51.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8008840 2021-03-09T09:35:51.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:51.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017380 con 0x7f63d8005e00 2021-03-09T09:35:51.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:51.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:51.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:51.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:51.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:51.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:51.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:35:51.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:51.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:51.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:51.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:51.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:51.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:51.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:51.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:35:51.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:35:51.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:51.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:51.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a490 con 0x7f63d8005e00 2021-03-09T09:35:51.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:51.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:51.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ab90 con 0x7f63d8005e00 2021-03-09T09:35:51.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:35:51.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:51.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7811 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc02dfe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:51.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:35:51.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.897+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:51.913 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:51.898+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21273 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:51.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:35:51.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.899+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:51.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.899+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:51.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.899+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:51.915 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:51.915 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:51.915 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:51.915 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:51.916 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:51.916 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:51.916 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:51.916 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:51.917 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:51.917 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:51.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03f790 con 0x7f63d8005e00 2021-03-09T09:35:51.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.899+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:51.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.899+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:35:51.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:51.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63d8005e00 2021-03-09T09:35:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:52.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:52.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:53.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:53.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:53.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:53.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:54.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:54.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:54.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:54.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:54.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:54.890+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:35:55.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:55.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:55.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:55.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:56.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:56.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:56.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:35:56.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:35:56.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7811 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc02dfe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:35:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7811 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc02dfe0).stop 2021-03-09T09:35:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:35:56.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:35:56.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:35:56.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:35:56.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:56.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:35:56.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:56.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:35:56.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:35:56.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:35:56.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:56.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b350 con 0x7f63d8003400 2021-03-09T09:35:56.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:56.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8008840 2021-03-09T09:35:56.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:35:56.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006acd0 con 0x7f6400146050 2021-03-09T09:35:56.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:35:56.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:35:56.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.898+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:56.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:35:56.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:35:56.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:35:56.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:35:56.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:35:56.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:35:56.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:35:56.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:35:56.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:35:56.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.899+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:35:56.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:56.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:56.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:35:56.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:35:56.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:35:56.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:35:56.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:56.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:56.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:35:56.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:35:56.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:35:56.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005dbd0 con 0x7f6400146050 2021-03-09T09:35:56.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:35:56.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:56.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 secure :-1 s=READY pgs=7824 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f006df80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:35:56.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:35:56.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.900+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:35:56.915 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:35:56.901+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21309 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:35:56.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f6400146050 2021-03-09T09:35:56.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.902+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:35:56.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.902+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:35:56.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.902+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:35:56.916 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:35:56.916 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:35:56.917 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:56.917 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:35:56.917 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:35:56.917 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:35:56.918 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:35:56.918 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:35:56.918 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:35:56.918 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:35:56.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007be40 con 0x7f6400146050 2021-03-09T09:35:56.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.902+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:35:56.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.902+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:35:56.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:56.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:35:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:57.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:57.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:58.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:58.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:58.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:58.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:58.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:58.452+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:35:59.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:59.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:35:59.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:59.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:35:59.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:35:59.891+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:00.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:00.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:00.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:00.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:01.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:01.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:01.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:01.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:01.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 secure :-1 s=READY pgs=7824 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f006df80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:01.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 secure :-1 s=READY pgs=7824 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f006df80).stop 2021-03-09T09:36:01.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:01.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:01.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:01.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 unknown :-1 s=CLOSED pgs=7824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:01.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:36:01.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:01.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:01.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:01.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:01.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:01.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:01.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:01.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:36:01.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:01.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d8c0 con 0x7f63d8008840 2021-03-09T09:36:01.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:01.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce70 con 0x7f63d8005e00 2021-03-09T09:36:01.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:01.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:01.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:01.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:01.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:01.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:01.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:36:01.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:01.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:01.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:01.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:01.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.902+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:01.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:01.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:01.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:01.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:01.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:01.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:36:01.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:36:01.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:01.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:01.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00230b0 con 0x7f63d8005e00 2021-03-09T09:36:01.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:01.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:01.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016370 con 0x7f63d8005e00 2021-03-09T09:36:01.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:36:01.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.903+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:01.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.904+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:36:01.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.904+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:01.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.904+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7840 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:01.918 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:01.905+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21350 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:01.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048a80 con 0x7f63d8005e00 2021-03-09T09:36:01.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.905+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:01.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.905+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:01.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.905+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:01.920 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:01.920 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:01.920 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:01.920 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:01.921 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:01.921 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:01.921 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:01.921 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:01.921 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:01.922 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:01.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63d8005e00 2021-03-09T09:36:01.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.906+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:01.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.906+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:36:01.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:01.908+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048a80 con 0x7f63d8005e00 2021-03-09T09:36:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:02.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:02.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:03.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:03.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:03.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:03.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:04.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:04.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:04.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:04.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:04.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:04.892+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:05.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:05.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:05.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:05.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:06.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:06.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:06.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:06.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:06.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7840 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:06.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7840 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003c20).stop 2021-03-09T09:36:06.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:06.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:06.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:06.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7840 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:06.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:36:06.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:06.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:06.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:06.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:06.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:06.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:06.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:06.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f5b0 con 0x7f63d8003400 2021-03-09T09:36:06.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:06.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d000 con 0x7f63d8008840 2021-03-09T09:36:06.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:06.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006acd0 con 0x7f6400146050 2021-03-09T09:36:06.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:06.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:06.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.905+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:06.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:06.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.906+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:06.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.906+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:06.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.906+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:36:06.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.906+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:06.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.906+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:06.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.907+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:06.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.907+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:06.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.907+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:06.925 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:06.908+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21395 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:06.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:06.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:06.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:06.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:06.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:06.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:36:06.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:36:06.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:06.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:06.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ab10 con 0x7f6400146050 2021-03-09T09:36:06.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:06.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a490 con 0x7f6400146050 2021-03-09T09:36:06.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:36:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:36:06.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 secure :-1 s=READY pgs=7860 cs=0 l=1 rev1=1 rx=0x7f63e005dda0 tx=0x7f63fc052fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:06.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:36:06.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.910+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:06.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.910+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:06.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.910+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:06.936 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:06.936 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:06.937 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:06.938 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:06.939 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:06.939 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:06.940 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:06.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:36:06.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.912+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:06.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.912+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:36:06.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:06.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f6400146050 2021-03-09T09:36:07.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:07.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:07.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:07.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:08.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:08.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:08.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:08.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:08.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:08.453+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:09.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:09.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:09.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:09.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:09.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:09.893+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:10.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:10.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:10.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:10.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:11.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:11.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:11.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:36:11.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:11.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 secure :-1 s=READY pgs=7860 cs=0 l=1 rev1=1 rx=0x7f63e005dda0 tx=0x7f63fc052fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:11.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 secure :-1 s=READY pgs=7860 cs=0 l=1 rev1=1 rx=0x7f63e005dda0 tx=0x7f63fc052fd0).stop 2021-03-09T09:36:11.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:11.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:11.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:11.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 unknown :-1 s=CLOSED pgs=7860 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:11.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:36:11.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:11.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:11.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:11.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ea10 con 0x7f63d8003400 2021-03-09T09:36:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:36:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bc00 con 0x7f63d8005e00 2021-03-09T09:36:11.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:11.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:11.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:36:11.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:11.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.909+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:11.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:11.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:11.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:11.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:11.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:11.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:11.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:11.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:11.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:36:11.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:36:11.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:11.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:11.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00018e0 con 0x7f63d8005e00 2021-03-09T09:36:11.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:11.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:11.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00230b0 con 0x7f63d8005e00 2021-03-09T09:36:11.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:36:11.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.910+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:11.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.911+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7867 cs=0 l=1 rev1=1 rx=0x7f63f4005230 tx=0x7f63f4003650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:11.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.911+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:36:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.911+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:11.927 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:11.911+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21434 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:11.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400de80 con 0x7f63d8005e00 2021-03-09T09:36:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.915+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.915+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.915+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:11.928 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:11.929 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:11.930 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:11.930 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:11.930 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:11.930 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:11.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400d9c0 con 0x7f63d8005e00 2021-03-09T09:36:11.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.915+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:11.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048a80 con 0x7f63d8005e00 2021-03-09T09:36:11.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:11.915+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:36:12.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:12.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:12.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:12.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:13.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:13.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:13.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:13.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:14.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:14.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:14.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:14.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:14.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:14.894+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:15.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:15.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:15.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:15.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:16.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:16.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:16.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:16.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:16.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7867 cs=0 l=1 rev1=1 rx=0x7f63f4005230 tx=0x7f63f4003650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:16.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7867 cs=0 l=1 rev1=1 rx=0x7f63f4005230 tx=0x7f63f4003650).stop 2021-03-09T09:36:16.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:16.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:16.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:16.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:16.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:36:16.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:16.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:16.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:16.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:16.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:16.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:16.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:16.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d8c0 con 0x7f63d8003400 2021-03-09T09:36:16.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:16.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00161f0 con 0x7f63d8008840 2021-03-09T09:36:16.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:16.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bc90 con 0x7f6400146050 2021-03-09T09:36:16.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:16.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:16.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:16.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:16.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.912+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:16.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:16.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.913+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:36:16.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.913+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:16.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.913+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:16.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.913+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:16.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.913+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:16.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.913+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:16.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:16.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:16.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:16.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:16.925 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21470 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:16.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:16.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:36:16.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:36:16.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:16.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:16.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ea30 con 0x7f6400146050 2021-03-09T09:36:16.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:16.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:16.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:36:16.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:36:16.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.914+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:16.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.915+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 secure :-1 s=READY pgs=7883 cs=0 l=1 rev1=1 rx=0x7f63f0023220 tx=0x7f63f000dee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:16.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.915+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:36:16.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.915+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:16.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:36:16.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.916+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:16.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.916+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:16.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.916+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:16.930 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:16.930 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:16.931 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:16.931 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:16.931 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:16.931 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:16.931 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:16.932 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:16.932 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:16.932 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:16.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:36:16.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.918+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:16.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.918+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:36:16.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:16.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:36:17.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:17.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:17.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:17.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:18.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:18.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:18.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:18.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:18.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:18.455+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:19.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:19.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:19.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:19.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:19.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:19.895+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:20.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:20.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:20.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:20.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:21.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:21.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:21.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:21.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:21.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 secure :-1 s=READY pgs=7883 cs=0 l=1 rev1=1 rx=0x7f63f0023220 tx=0x7f63f000dee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:21.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 secure :-1 s=READY pgs=7883 cs=0 l=1 rev1=1 rx=0x7f63f0023220 tx=0x7f63f000dee0).stop 2021-03-09T09:36:21.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:21.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:21.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:21.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=7883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:21.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:36:21.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:21.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:21.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:21.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:21.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:21.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:21.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:21.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010d20 con 0x7f63d8003400 2021-03-09T09:36:21.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:21.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8008840 2021-03-09T09:36:21.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:21.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e6c0 con 0x7f63d8005e00 2021-03-09T09:36:21.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:21.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:21.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:21.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:21.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:21.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:21.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:36:21.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:21.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:21.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:21.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:21.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:21.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:21.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:21.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:21.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:21.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:21.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:36:21.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:36:21.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:21.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:21.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:36:21.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:21.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:21.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ab10 con 0x7f63d8005e00 2021-03-09T09:36:21.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.917+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:36:21.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.917+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:21.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.917+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7893 cs=0 l=1 rev1=1 rx=0x7f63fc016e90 tx=0x7f63fc00b5d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:21.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.917+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:36:21.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.917+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:21.930 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:21.918+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21511 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:21.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d370 con 0x7f63d8005e00 2021-03-09T09:36:21.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.919+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:21.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.919+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:21.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.919+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:21.932 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:21.932 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:21.932 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:21.933 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:21.933 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:21.933 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:21.933 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:21.933 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:21.934 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:21.934 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:21.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f63d8005e00 2021-03-09T09:36:21.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.919+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:21.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.919+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:36:21.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:21.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T09:36:22.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:22.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:22.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:22.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:23.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:23.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:23.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:23.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:24.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:24.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:24.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:24.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:24.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:24.896+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:25.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:25.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:25.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:25.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:26.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:26.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:26.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:36:26.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:26.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7893 cs=0 l=1 rev1=1 rx=0x7f63fc016e90 tx=0x7f63fc00b5d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:26.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7893 cs=0 l=1 rev1=1 rx=0x7f63fc016e90 tx=0x7f63fc00b5d0).stop 2021-03-09T09:36:26.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:26.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:26.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:26.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7893 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:26.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:36:26.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:26.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:26.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:26.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:26.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:26.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:26.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:26.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:36:26.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:26.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce70 con 0x7f63d8008840 2021-03-09T09:36:26.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:26.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400146050 2021-03-09T09:36:26.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:26.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.918+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:26.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.919+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:26.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.919+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:26.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.919+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:26.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.919+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:26.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.919+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:36:26.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.919+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:26.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.919+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:26.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:26.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:26.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:26.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:26.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:26.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:26.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:26.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:26.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:36:26.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:36:26.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:26.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:26.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b300 con 0x7f6400146050 2021-03-09T09:36:26.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:26.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:26.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:36:26.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:36:26.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.920+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:26.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 secure :-1 s=READY pgs=7911 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc012ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:26.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.921+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:36:26.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.921+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:26.945 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:26.921+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21549 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:26.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d370 con 0x7f6400146050 2021-03-09T09:36:26.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.922+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:26.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.922+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:26.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.922+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:26.947 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:26.947 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:26.947 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:26.947 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:26.947 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:26.948 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:26.948 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:26.948 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:26.948 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:26.949 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:26.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d190 con 0x7f6400146050 2021-03-09T09:36:26.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.922+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:26.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.922+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:36:26.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:26.923+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021460 con 0x7f6400146050 2021-03-09T09:36:27.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:27.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:27.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:27.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:28.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:28.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:28.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:28.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:28.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:28.456+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:29.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:29.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:29.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:29.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:29.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:29.897+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:30.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:30.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:30.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:30.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:31.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:31.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:31.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:36:31.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:31.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 secure :-1 s=READY pgs=7911 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc012ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:31.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 secure :-1 s=READY pgs=7911 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc012ed0).stop 2021-03-09T09:36:31.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:31.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:31.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:31.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001c150 unknown :-1 s=CLOSED pgs=7911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:31.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:36:31.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:31.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:31.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:31.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:31.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:31.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:31.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:31.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010d20 con 0x7f63d8003400 2021-03-09T09:36:31.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:31.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8008840 2021-03-09T09:36:31.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:31.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:36:31.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:31.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:31.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:31.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:31.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:31.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.923+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:31.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.923+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:36:31.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.923+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:31.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.923+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:31.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:31.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:31.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:31.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:31.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:31.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:31.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:31.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:31.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:36:31.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:36:31.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:31.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:31.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T09:36:31.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:31.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:31.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ea30 con 0x7f63d8005e00 2021-03-09T09:36:31.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:36:31.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.924+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:31.938 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:31.925+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21586 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:31.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.925+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7926 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63f0013b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:31.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.925+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:36:31.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.925+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:31.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T09:36:31.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.926+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:31.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.926+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:31.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.926+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:31.941 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:31.941 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:31.941 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:31.941 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:31.942 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:31.942 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:31.942 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:31.942 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:31.942 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:31.943 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:31.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T09:36:31.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.928+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:31.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027d00 con 0x7f63d8005e00 2021-03-09T09:36:31.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:31.928+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:36:32.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:32.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:32.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:32.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:33.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:33.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:33.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:33.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:34.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:34.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:34.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:34.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:34.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:34.898+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:35.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:35.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:35.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:35.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:36.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:36.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:36.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:36:36.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:36.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7926 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63f0013b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:36.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7926 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63f0013b10).stop 2021-03-09T09:36:36.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:36.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:36.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:36.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7926 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:36.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:36:36.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:36.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:36.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:36.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:36.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:36.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:36.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:36.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f8e0 con 0x7f63d8003400 2021-03-09T09:36:36.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:36.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ab30 con 0x7f63d8008840 2021-03-09T09:36:36.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:36.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c900 con 0x7f6400146050 2021-03-09T09:36:36.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:36.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:36.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:36.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:36.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:36.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:36.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:36:36.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:36.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.926+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:36.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:36.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:36.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:36.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:36.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:36.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:36.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:36.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:36:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:36:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:36.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00230b0 con 0x7f6400146050 2021-03-09T09:36:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.927+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:36:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.928+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:36:36.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.928+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:36.941 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:36.928+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21626 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 secure :-1 s=READY pgs=7941 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.929+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:36:36.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.929+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:36.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f6400146050 2021-03-09T09:36:36.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.930+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:36.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.930+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:36.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.930+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:36.944 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:36.945 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:36.946 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:36.947 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:36.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f6400146050 2021-03-09T09:36:36.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.932+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:36.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.932+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:36:36.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:36.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00284b0 con 0x7f6400146050 2021-03-09T09:36:37.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:37.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:37.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:37.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:38.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:38.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:38.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:38.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:38.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:38.457+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:39.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:39.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:39.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:39.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:39.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:39.899+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:40.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:40.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:40.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:40.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:41.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:41.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:41.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:36:41.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:41.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 secure :-1 s=READY pgs=7941 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:41.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 secure :-1 s=READY pgs=7941 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006c30).stop 2021-03-09T09:36:41.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:41.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:41.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:41.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=7941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:41.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:36:41.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:41.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:41.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:41.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:41.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.928+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:41.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:41.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:41.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63d8003400 2021-03-09T09:36:41.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:41.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce70 con 0x7f63d8008840 2021-03-09T09:36:41.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:41.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T09:36:41.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:41.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:41.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:41.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:41.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:41.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.930+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:41.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.930+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:36:41.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.930+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:41.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.930+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:41.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:41.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:41.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:41.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:41.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:41.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:41.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:41.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:41.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:36:41.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:36:41.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:41.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:41.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001750 con 0x7f63d8005e00 2021-03-09T09:36:41.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:41.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:41.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00018e0 con 0x7f63d8005e00 2021-03-09T09:36:41.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:36:41.944 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21665 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:41.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.931+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:41.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.932+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:36:41.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.932+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7959 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4022500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:41.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.932+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:41.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005ad0 con 0x7f63d8005e00 2021-03-09T09:36:41.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.933+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:41.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.933+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:41.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.933+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:41.947 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:41.947 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:41.947 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:41.948 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:41.948 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:41.948 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:41.948 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:41.948 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:41.949 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:41.949 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:41.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f63d8005e00 2021-03-09T09:36:41.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.934+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:41.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.934+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:36:41.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:41.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005ad0 con 0x7f63d8005e00 2021-03-09T09:36:42.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:42.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:42.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:42.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:43.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:43.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:43.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:43.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:44.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:44.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:44.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:44.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:44.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:44.900+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:45.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:45.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:45.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:45.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:46.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:46.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:46.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:36:46.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:46.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7959 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4022500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:46.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7959 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4022500).stop 2021-03-09T09:36:46.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:46.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:46.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:46.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7959 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:46.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:36:46.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:46.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:46.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:46.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:46.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:46.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:46.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:46.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:36:46.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:46.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017380 con 0x7f63d8008840 2021-03-09T09:36:46.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:46.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f6400146050 2021-03-09T09:36:46.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:46.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:46.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.932+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:46.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:46.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:46.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:46.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:36:46.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:46.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:46.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:46.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:46.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:46.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:46.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:46.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:46.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:46.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.934+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:46.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:36:46.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:36:46.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:46.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:46.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:36:46.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:46.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:46.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001750 con 0x7f6400146050 2021-03-09T09:36:46.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:36:46.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:46.954 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21703 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:46.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:36:46.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:46.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025e10 secure :-1 s=READY pgs=7967 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404b4c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f6400146050 2021-03-09T09:36:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.938+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:46.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.938+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.938+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:46.957 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:46.958 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:46.958 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:46.958 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:46.958 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:46.959 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:46.959 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:46.959 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:46.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022a80 con 0x7f6400146050 2021-03-09T09:36:46.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.938+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.938+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:36:46.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:46.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022bf0 con 0x7f6400146050 2021-03-09T09:36:47.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:47.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:47.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:47.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:48.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:48.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:48.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:48.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:48.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:48.459+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:49.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:49.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:49.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:49.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:49.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:49.901+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:50.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:50.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:50.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:50.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:51.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:51.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025e10 secure :-1 s=READY pgs=7967 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404b4c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025e10 secure :-1 s=READY pgs=7967 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404b4c0).stop 2021-03-09T09:36:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:51.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0025e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:51.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0025e10 unknown :-1 s=CLOSED pgs=7967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:51.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:36:51.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:51.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:51.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:51.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:51.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:51.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:51.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:51.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63d8003400 2021-03-09T09:36:51.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:51.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029cf0 con 0x7f63d8008840 2021-03-09T09:36:51.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:51.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8005e00 2021-03-09T09:36:51.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:51.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:51.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.935+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:51.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:51.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:51.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:51.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:36:51.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:51.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:51.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:51.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:51.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:51.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:51.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:51.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.936+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:51.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:51.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:51.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:36:51.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:36:51.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:51.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:51.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:36:51.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:51.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:51.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:36:51.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:36:51.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:51.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7975 cs=0 l=1 rev1=1 rx=0x7f63f4003620 tx=0x7f63f40223d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:51.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:36:51.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.937+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:51.951 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:51.938+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21744 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:51.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400ee60 con 0x7f63d8005e00 2021-03-09T09:36:51.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.938+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:51.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.938+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:51.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.938+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:51.953 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:51.953 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:51.953 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:51.953 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:51.954 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:51.954 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:51.954 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:51.954 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:51.954 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:51.955 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:51.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f401a270 con 0x7f63d8005e00 2021-03-09T09:36:51.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.938+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:51.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.938+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:36:51.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:51.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022a80 con 0x7f63d8005e00 2021-03-09T09:36:52.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:52.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:52.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:52.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:53.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:53.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:53.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:53.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:54.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:54.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:54.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:54.902+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:36:55.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:55.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:55.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:55.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:56.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:56.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:36:56.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:36:56.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7975 cs=0 l=1 rev1=1 rx=0x7f63f4003620 tx=0x7f63f40223d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:36:56.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=7975 cs=0 l=1 rev1=1 rx=0x7f63f4003620 tx=0x7f63f40223d0).stop 2021-03-09T09:36:56.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:56.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:36:56.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:36:56.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=7975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:36:56.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:36:56.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:56.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:36:56.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:56.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:36:56.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:36:56.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:36:56.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:56.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006be00 con 0x7f63d8003400 2021-03-09T09:36:56.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:56.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c220 con 0x7f63d8008840 2021-03-09T09:36:56.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:36:56.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:36:56.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:36:56.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:36:56.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:56.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:36:56.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:36:56.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:36:56.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:36:56.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:36:56.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.939+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:36:56.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:36:56.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:36:56.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:36:56.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:36:56.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:56.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:56.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:36:56.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:36:56.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:36:56.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:36:56.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:56.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:56.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:36:56.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:36:56.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:36:56.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:36:56.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:36:56.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.940+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:56.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.941+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 secure :-1 s=READY pgs=7991 cs=0 l=1 rev1=1 rx=0x7f63f4009960 tx=0x7f63f4003ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:36:56.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.941+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:36:56.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.941+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:36:56.961 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:36:56.942+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21780 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:36:56.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4046b80 con 0x7f6400146050 2021-03-09T09:36:56.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.943+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:36:56.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.943+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:36:56.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.943+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:36:56.963 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:36:56.963 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:36:56.963 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:56.963 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:36:56.964 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:36:56.964 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:36:56.964 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:36:56.964 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:36:56.965 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:36:56.965 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:36:56.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046b80 con 0x7f6400146050 2021-03-09T09:36:56.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.947+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:36:56.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.947+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:36:56.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:56.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404c310 con 0x7f6400146050 2021-03-09T09:36:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:57.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:57.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:58.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:58.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:58.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:58.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:58.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:58.460+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:36:59.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:59.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:36:59.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:59.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:36:59.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:36:59.903+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:00.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:00.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:00.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:00.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:01.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:01.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:01.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 secure :-1 s=READY pgs=7991 cs=0 l=1 rev1=1 rx=0x7f63f4009960 tx=0x7f63f4003ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:01.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 secure :-1 s=READY pgs=7991 cs=0 l=1 rev1=1 rx=0x7f63f4009960 tx=0x7f63f4003ad0).stop 2021-03-09T09:37:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 unknown :-1 s=CLOSED pgs=7991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:37:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016f50 con 0x7f63d8003400 2021-03-09T09:37:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c900 con 0x7f63d8008840 2021-03-09T09:37:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T09:37:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.943+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.944+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:37:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.944+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.944+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.944+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.944+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:01.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.944+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:37:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:37:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015640 con 0x7f63d8005e00 2021-03-09T09:37:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00230b0 con 0x7f63d8005e00 2021-03-09T09:37:01.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:37:01.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:01.958 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:01.945+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21821 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:01.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.946+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8010 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f0013bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:01.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.946+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:37:01.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.946+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:01.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.948+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f63d8005e00 2021-03-09T09:37:01.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.948+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:01.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.948+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:01.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.948+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:01.961 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:01.963 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:01.963 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:01.963 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:01.963 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:01.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f63d8005e00 2021-03-09T09:37:01.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.949+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:01.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.949+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:37:01.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:01.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T09:37:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:02.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:02.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:02.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:03.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:03.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:03.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:03.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:04.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:04.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:04.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:04.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:04.904+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:05.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:05.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:05.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:05.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:06.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:06.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:06.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:06.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:06.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8010 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f0013bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:06.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8010 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f0013bf0).stop 2021-03-09T09:37:06.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:06.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:06.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:06.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:06.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:37:06.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:06.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:06.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:06.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:06.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:06.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:06.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:06.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8003400 2021-03-09T09:37:06.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:06.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63d8008840 2021-03-09T09:37:06.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:06.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b6c0 con 0x7f6400146050 2021-03-09T09:37:06.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:06.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:06.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:06.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:06.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.946+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:06.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.947+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:06.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.947+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:37:06.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.947+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:06.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.947+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:06.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:06.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:06.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:06.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:37:06.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:37:06.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:06.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:06.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b440 con 0x7f6400146050 2021-03-09T09:37:06.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:06.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:06.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015640 con 0x7f6400146050 2021-03-09T09:37:06.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:37:06.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.948+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:06.961 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:06.949+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21859 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:06.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.949+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e720 secure :-1 s=READY pgs=8021 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0036e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:06.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.949+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:37:06.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.949+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:06.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001c4a0 con 0x7f6400146050 2021-03-09T09:37:06.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.950+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:06.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.950+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:06.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.950+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:06.964 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:06.964 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:06.964 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:06.966 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:06.966 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:06.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.952+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00036b0 con 0x7f6400146050 2021-03-09T09:37:06.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.952+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:06.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.952+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:37:06.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:06.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0003820 con 0x7f6400146050 2021-03-09T09:37:07.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:07.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:07.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:07.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:08.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:08.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:08.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:08.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:08.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:08.461+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:09.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:09.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:09.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:09.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:09.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:09.905+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:10.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:10.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:10.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:10.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:11.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:11.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:11.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:11.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:11.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e720 secure :-1 s=READY pgs=8021 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0036e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:11.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e720 secure :-1 s=READY pgs=8021 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0036e00).stop 2021-03-09T09:37:11.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:11.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:11.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001e720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:11.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001e720 unknown :-1 s=CLOSED pgs=8021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:11.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:37:11.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0077c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:11.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:11.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:11.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:11.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:11.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:11.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:11.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006be00 con 0x7f63d8003400 2021-03-09T09:37:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067510 con 0x7f63d8008840 2021-03-09T09:37:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T09:37:11.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.950+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:11.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.951+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:11.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.951+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:11.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:11.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:11.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.951+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0077c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:11.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.951+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:37:11.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.951+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:11.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.951+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:11.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.952+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:11.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.952+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:11.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.952+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:11.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:11.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:11.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:11.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:11.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:11.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:37:11.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:37:11.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:11.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:37:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:37:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:37:11.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:11.965 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21898 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:11.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:37:11.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0077c20 secure :-1 s=READY pgs=8037 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f404bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.953+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f63d8005e00 2021-03-09T09:37:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.955+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.955+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.955+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:11.969 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:11.969 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:11.969 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:11.969 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:11.970 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f63d8005e00 2021-03-09T09:37:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.956+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.956+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:37:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:11.958+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402cc40 con 0x7f63d8005e00 2021-03-09T09:37:12.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:12.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:12.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:12.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:13.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:13.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:13.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:13.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:14.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:14.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:14.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:14.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:14.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:14.906+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:15.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:15.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:15.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:16.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:16.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:16.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0077c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:16.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0077c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:16.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0077c20 secure :-1 s=READY pgs=8037 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f404bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:16.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0077c20 secure :-1 s=READY pgs=8037 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f404bfd0).stop 2021-03-09T09:37:16.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:16.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:16.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0077c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:16.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0077c20 unknown :-1 s=CLOSED pgs=8037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:16.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:37:16.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:16.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:16.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:16.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:16.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:16.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:16.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:16.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024b30 con 0x7f63d8003400 2021-03-09T09:37:16.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:16.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b6c0 con 0x7f63d8008840 2021-03-09T09:37:16.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:16.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f6400146050 2021-03-09T09:37:16.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:16.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:16.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.954+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:16.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.955+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:16.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.955+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:16.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:16.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.956+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:37:16.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.956+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:16.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.956+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:16.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.956+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:16.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.956+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:16.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.956+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:16.973 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:16.957+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21936 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:16.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:16.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:16.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:16.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:16.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:16.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:37:16.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:37:16.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:16.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:16.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d9b0 con 0x7f6400146050 2021-03-09T09:37:16.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:16.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:16.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b440 con 0x7f6400146050 2021-03-09T09:37:16.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:37:16.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.958+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:16.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 secure :-1 s=READY pgs=8054 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f002d300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:16.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.960+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:37:16.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.960+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:16.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f6400146050 2021-03-09T09:37:16.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.961+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:16.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.961+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:16.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.961+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:16.979 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:16.979 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:16.979 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:16.979 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:16.980 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:16.980 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:16.980 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:16.980 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:16.980 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:16.981 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:16.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f6400146050 2021-03-09T09:37:16.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.963+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:16.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.963+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:37:16.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:16.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:37:17.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:17.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:17.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:18.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:18.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:18.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:18.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:18.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:18.463+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:19.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:19.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:19.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:19.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:19.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:19.907+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:20.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:20.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:20.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:20.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:21.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:21.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:21.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:21.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:21.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 secure :-1 s=READY pgs=8054 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f002d300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:21.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 secure :-1 s=READY pgs=8054 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f002d300).stop 2021-03-09T09:37:21.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:21.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:21.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:21.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 unknown :-1 s=CLOSED pgs=8054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:21.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:37:21.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:21.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:21.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:21.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:21.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:21.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:21.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:21.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63d8003400 2021-03-09T09:37:21.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:37:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:21.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006be00 con 0x7f63d8005e00 2021-03-09T09:37:21.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.958+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:21.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.958+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:21.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.958+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:21.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.959+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:21.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.959+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:21.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:21.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.960+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:37:21.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.960+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:21.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.960+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:21.973 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:21.960+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 21973 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:21.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.961+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:21.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.961+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:21.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.961+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:21.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:21.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:21.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:21.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:21.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:21.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:37:21.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:37:21.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:21.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:21.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015640 con 0x7f63d8005e00 2021-03-09T09:37:21.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:21.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:21.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:37:21.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:37:21.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:21.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8068 cs=0 l=1 rev1=1 rx=0x7f63f404c040 tx=0x7f63f40618d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:21.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:37:21.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.962+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:21.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022440 con 0x7f63d8005e00 2021-03-09T09:37:21.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.966+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:21.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.966+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:21.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.966+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:21.980 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:21.982 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:21.982 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:21.982 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:21.983 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:21.983 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:21.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63d8005e00 2021-03-09T09:37:21.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.968+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:21.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.968+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:37:21.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:21.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022440 con 0x7f63d8005e00 2021-03-09T09:37:22.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:22.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:22.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:22.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:23.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:23.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:24.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:24.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:24.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:24.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:24.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:24.908+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:25.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:25.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:25.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:25.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:26.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:26.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:26.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.960+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:26.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.960+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:26.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8068 cs=0 l=1 rev1=1 rx=0x7f63f404c040 tx=0x7f63f40618d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:26.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8068 cs=0 l=1 rev1=1 rx=0x7f63f404c040 tx=0x7f63f40618d0).stop 2021-03-09T09:37:26.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:26.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:26.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:26.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:26.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:37:26.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:26.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:26.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:26.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:26.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:26.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:26.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:26.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e0d0 con 0x7f63d8003400 2021-03-09T09:37:26.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:26.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017910 con 0x7f63d8008840 2021-03-09T09:37:26.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:26.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f6400146050 2021-03-09T09:37:26.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:26.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:26.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:26.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:26.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:26.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:26.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:37:26.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:26.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.961+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:26.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.962+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:26.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.962+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:26.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.962+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:26.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:26.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:26.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:26.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:26.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:26.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:37:26.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:37:26.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:26.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:26.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:37:26.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:26.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:26.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b300 con 0x7f6400146050 2021-03-09T09:37:26.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:37:26.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:26.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 secure :-1 s=READY pgs=8074 cs=0 l=1 rev1=1 rx=0x7f63e002c0f0 tx=0x7f63fc013f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:26.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:37:26.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:26.976 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:26.963+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22013 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:26.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T09:37:26.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.964+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:26.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.964+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:26.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.964+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:26.978 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:26.978 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:26.979 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:26.979 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:26.979 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:26.979 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:26.979 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:26.980 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:26.980 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:26.980 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:26.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T09:37:26.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.964+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:26.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.964+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:37:26.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:26.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021460 con 0x7f6400146050 2021-03-09T09:37:27.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:27.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:27.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:27.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:28.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:28.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:28.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:28.464+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:29.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:29.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:29.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:29.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:29.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:29.909+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:30.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:30.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:30.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:30.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:31.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:31.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:31.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:31.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:31.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 secure :-1 s=READY pgs=8074 cs=0 l=1 rev1=1 rx=0x7f63e002c0f0 tx=0x7f63fc013f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:31.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 secure :-1 s=READY pgs=8074 cs=0 l=1 rev1=1 rx=0x7f63e002c0f0 tx=0x7f63fc013f90).stop 2021-03-09T09:37:31.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:31.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:31.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:31.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f9c0 unknown :-1 s=CLOSED pgs=8074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:31.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:37:31.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:31.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:31.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:31.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:31.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:31.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:31.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:31.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024b30 con 0x7f63d8003400 2021-03-09T09:37:31.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:31.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b6c0 con 0x7f63d8008840 2021-03-09T09:37:31.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:31.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017b10 con 0x7f63d8005e00 2021-03-09T09:37:31.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:31.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:31.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:31.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:31.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:37:31.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:31.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:31.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:31.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:31.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:31.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.964+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:31.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:31.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:31.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:31.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:31.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:31.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:31.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:37:31.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:37:31.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:31.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:31.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:37:31.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:31.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:31.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015640 con 0x7f63d8005e00 2021-03-09T09:37:31.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:37:31.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.965+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.966+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:37:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.966+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:31.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.966+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8085 cs=0 l=1 rev1=1 rx=0x7f63f4046c10 tx=0x7f63f401b350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:31.978 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:31.967+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22052 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:31.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401ae70 con 0x7f63d8005e00 2021-03-09T09:37:31.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.967+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:31.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.967+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:31.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.967+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:31.980 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:31.980 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:31.981 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:31.981 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:31.981 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:31.981 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:31.982 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:31.982 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:31.982 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:31.982 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:31.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63d8005e00 2021-03-09T09:37:31.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.967+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:31.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.967+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:37:31.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:31.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404ddb0 con 0x7f63d8005e00 2021-03-09T09:37:32.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:32.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:32.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:32.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:33.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:33.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:34.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:34.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:34.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:34.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:34.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:34.910+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:35.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:35.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:35.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:35.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:36.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:36.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:36.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:36.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8085 cs=0 l=1 rev1=1 rx=0x7f63f4046c10 tx=0x7f63f401b350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:36.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8085 cs=0 l=1 rev1=1 rx=0x7f63f4046c10 tx=0x7f63f401b350).stop 2021-03-09T09:37:36.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:36.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:36.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:36.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:36.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:37:36.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:36.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:36.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:36.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:36.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:36.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:36.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:36.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:37:36.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:36.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077680 con 0x7f63d8008840 2021-03-09T09:37:36.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:36.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00249d0 con 0x7f6400146050 2021-03-09T09:37:36.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:36.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:36.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.966+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:36.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.967+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:36.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.967+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:36.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.967+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:36.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.967+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:37:36.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.967+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:36.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.967+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:36.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.967+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:36.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.967+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:36.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.967+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:36.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:36.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:36.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:36.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:36.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:36.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:37:36.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:37:36.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:37.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:37.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001e40 con 0x7f6400146050 2021-03-09T09:37:37.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:37.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:37.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d9b0 con 0x7f6400146050 2021-03-09T09:37:37.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:37:37.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.968+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:37.001 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:36.969+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22090 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:37.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010240 secure :-1 s=READY pgs=8104 cs=0 l=1 rev1=1 rx=0x7f63e00245a0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:37.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.970+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:37:37.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.970+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:37.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f6400146050 2021-03-09T09:37:37.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.970+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:37.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.970+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:37.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.970+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:37.004 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:37.004 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:37.006 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:37.006 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:37.006 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:37.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.972+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f6400146050 2021-03-09T09:37:37.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.972+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:37.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.972+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:37:37.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:36.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:37:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:37.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:37.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:37.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:38.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:38.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:38.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:38.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:38.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:38.464+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:39.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:39.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:39.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:39.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:39.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:39.910+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:40.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:40.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:40.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:40.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:41.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:41.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:41.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0010240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:41.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0010240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:41.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010240 secure :-1 s=READY pgs=8104 cs=0 l=1 rev1=1 rx=0x7f63e00245a0 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:41.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010240 secure :-1 s=READY pgs=8104 cs=0 l=1 rev1=1 rx=0x7f63e00245a0 tx=0x7f63f004db20).stop 2021-03-09T09:37:41.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:41.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:41.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0010240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:41.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0010240 unknown :-1 s=CLOSED pgs=8104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:41.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:37:41.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:41.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:41.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:41.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:41.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:41.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:41.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:41.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T09:37:41.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:41.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024040 con 0x7f63d8008840 2021-03-09T09:37:41.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:41.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T09:37:41.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:41.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:41.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:41.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:41.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:41.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:41.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:37:41.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:41.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.970+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:41.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:41.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:41.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:41.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:41.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:41.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:41.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:41.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:41.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:37:41.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:37:41.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:41.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:41.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ebd0 con 0x7f63d8005e00 2021-03-09T09:37:41.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.971+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:41.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.972+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:41.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.972+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001e40 con 0x7f63d8005e00 2021-03-09T09:37:41.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.972+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:37:41.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.972+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:41.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.972+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:37:41.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.972+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:41.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.972+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8115 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63f0013c40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:41.988 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:41.972+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22129 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:41.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f63d8005e00 2021-03-09T09:37:41.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.975+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:41.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.975+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:41.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.975+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:41.990 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:41.990 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:41.990 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:41.990 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:41.991 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:41.991 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:41.991 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:41.991 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:41.991 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:41.992 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:41.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f63d8005e00 2021-03-09T09:37:41.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.978+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:41.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.978+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:37:41.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:41.979+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T09:37:42.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:42.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:42.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:42.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:43.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:43.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:43.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:43.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:44.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:44.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:44.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:44.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:44.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:44.911+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:45.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:45.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:45.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:45.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:46.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:46.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:46.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:37:46.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:46.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8115 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63f0013c40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:46.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8115 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63f0013c40).stop 2021-03-09T09:37:46.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:46.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:46.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:46.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:46.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:37:46.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:46.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:46.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:46.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:46.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:46.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:46.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:46.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b6c0 con 0x7f63d8003400 2021-03-09T09:37:46.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:46.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8008840 2021-03-09T09:37:46.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:46.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:37:46.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:46.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:46.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.973+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:46.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.974+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:46.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.974+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:46.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.974+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:37:46.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.974+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:46.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.974+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:46.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.974+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:46.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.974+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:46.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.974+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:46.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.974+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:46.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:46.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:46.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:46.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:46.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:46.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:37:46.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:37:46.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:46.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:46.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015640 con 0x7f6400146050 2021-03-09T09:37:46.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:46.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:46.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:37:46.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:37:46.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.975+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:46.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.976+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006b700 secure :-1 s=READY pgs=8132 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc02e000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:46.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.976+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:37:46.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.976+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:46.999 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:46.976+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22167 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:47.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f6400146050 2021-03-09T09:37:47.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.977+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:47.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.977+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:47.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.977+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:47.002 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:47.003 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:47.003 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:47.004 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:47.004 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:47.004 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:47.005 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:47.005 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:47.006 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:47.006 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:47.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f6400146050 2021-03-09T09:37:47.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.978+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:47.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.978+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:37:47.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:46.979+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:37:47.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:47.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:47.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:47.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:48.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:48.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:48.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:48.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:48.466+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:49.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:49.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:49.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:49.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:49.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:49.912+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:50.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:50.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:50.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:50.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:51.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:51.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:51.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.976+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:51.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.976+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:51.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.976+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006b700 secure :-1 s=READY pgs=8132 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc02e000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:51.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.976+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006b700 secure :-1 s=READY pgs=8132 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc02e000).stop 2021-03-09T09:37:51.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.976+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:51.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.976+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:51.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006b700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:51.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.976+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006b700 unknown :-1 s=CLOSED pgs=8132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:51.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.976+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:37:51.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:51.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:51.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:51.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:51.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:51.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:51.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:51.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63d8003400 2021-03-09T09:37:51.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:51.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cd00 con 0x7f63d8008840 2021-03-09T09:37:51.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:51.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024040 con 0x7f63d8005e00 2021-03-09T09:37:51.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:51.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:51.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:51.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:37:51.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:51.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:51.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:51.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:51.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:51.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:51.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:51.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.977+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:51.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:51.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:51.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:51.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:51.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:51.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:37:51.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:37:51.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:51.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:51.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001e40 con 0x7f63d8005e00 2021-03-09T09:37:51.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:51.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:51.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:37:51.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:37:51.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.978+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:51.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.979+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:37:51.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.979+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:51.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.979+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8141 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4005180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:51.991 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:51.980+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22206 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:51.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4046850 con 0x7f63d8005e00 2021-03-09T09:37:51.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.980+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:51.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.980+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:51.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.980+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:51.993 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:51.993 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:51.993 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:51.994 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:51.994 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:51.994 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:51.994 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:51.994 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:51.995 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046850 con 0x7f63d8005e00 2021-03-09T09:37:51.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.982+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:51.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.982+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:37:51.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:51.983+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400d1c0 con 0x7f63d8005e00 2021-03-09T09:37:52.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:52.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:52.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:52.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:53.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:53.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:53.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:54.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:54.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:54.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:54.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:54.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:54.913+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:37:55.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:55.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:55.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:55.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:56.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:56.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:56.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:37:56.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:37:56.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8141 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4005180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:37:56.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8141 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4005180).stop 2021-03-09T09:37:56.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:56.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:37:56.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:37:56.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:37:56.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:37:56.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:56.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:37:56.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:56.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:37:56.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:37:56.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:37:56.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:56.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067cd0 con 0x7f63d8003400 2021-03-09T09:37:56.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:56.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f5b0 con 0x7f63d8008840 2021-03-09T09:37:56.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:37:56.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400146050 2021-03-09T09:37:56.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:37:56.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:37:56.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:37:56.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:56.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.980+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:37:56.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.981+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:37:56.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.981+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:37:56.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.981+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:37:56.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.981+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:37:56.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:37:56.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:37:56.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:37:56.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:37:56.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:56.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:56.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:37:56.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:37:56.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:37:56.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:37:56.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:56.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:56.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001efb0 con 0x7f6400146050 2021-03-09T09:37:56.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:37:56.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:37:56.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.982+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ebd0 con 0x7f6400146050 2021-03-09T09:37:56.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:37:56.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:56.996 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22244 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:37:56.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 secure :-1 s=READY pgs=8160 cs=0 l=1 rev1=1 rx=0x7f63e0012a80 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:37:56.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:37:56.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:37:56.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:37:56.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:37:56.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:37:56.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.983+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:37:56.999 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:37:56.999 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:37:56.999 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:57.000 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:37:57.000 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:37:57.000 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:37:57.000 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:37:57.001 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:37:57.001 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:37:57.001 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:37:57.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.985+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:37:57.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.985+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:37:57.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.985+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:37:57.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:56.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:37:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:57.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:57.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:58.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:58.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:58.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:58.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:58.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:58.467+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:37:59.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:59.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:37:59.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:59.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:37:59.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:37:59.914+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:00.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:00.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:00.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:00.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:01.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:01.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:01.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:01.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:01.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.983+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 secure :-1 s=READY pgs=8160 cs=0 l=1 rev1=1 rx=0x7f63e0012a80 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:01.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.983+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 secure :-1 s=READY pgs=8160 cs=0 l=1 rev1=1 rx=0x7f63e0012a80 tx=0x7f63f0006610).stop 2021-03-09T09:38:01.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.983+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:01.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.983+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:01.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.983+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:01.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.983+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 unknown :-1 s=CLOSED pgs=8160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:01.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:38:01.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:01.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:01.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:01.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:01.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:01.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:01.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:01.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024240 con 0x7f63d8003400 2021-03-09T09:38:01.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:01.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b6c0 con 0x7f63d8008840 2021-03-09T09:38:01.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:01.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8005e00 2021-03-09T09:38:01.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:01.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:01.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:01.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:01.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:01.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:01.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:38:01.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:01.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:01.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:01.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:01.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.984+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:01.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:01.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:38:01.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:38:01.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:01.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:01.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a970 con 0x7f63d8005e00 2021-03-09T09:38:01.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:01.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:01.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001efb0 con 0x7f63d8005e00 2021-03-09T09:38:01.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:38:01.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:01.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:38:01.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:01.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.985+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8170 cs=0 l=1 rev1=1 rx=0x7f63e00653c0 tx=0x7f63f007ef40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:01.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T09:38:02.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.987+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:02.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.987+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:02.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.987+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:02.000 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:02.000 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:02.001 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:02.001 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:02.001 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:02.001 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:02.001 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:02.002 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:02.002 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:02.002 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:02.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T09:38:02.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.987+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:02.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.987+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:38:02.003 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:01.986+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22282 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:02.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:01.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f006d4d0 con 0x7f63d8005e00 2021-03-09T09:38:02.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:02.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:02.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:02.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:03.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:03.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:03.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:03.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:04.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:04.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:04.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:04.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:04.915+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:05.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:05.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:05.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:05.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:06.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:06.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:06.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:06.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:06.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8170 cs=0 l=1 rev1=1 rx=0x7f63e00653c0 tx=0x7f63f007ef40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:06.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8170 cs=0 l=1 rev1=1 rx=0x7f63e00653c0 tx=0x7f63f007ef40).stop 2021-03-09T09:38:06.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:06.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:06.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:06.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:06.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:38:06.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:06.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:06.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:06.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:06.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:06.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:06.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:06.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d240 con 0x7f63d8003400 2021-03-09T09:38:06.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:06.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8008840 2021-03-09T09:38:06.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:06.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f5b0 con 0x7f6400146050 2021-03-09T09:38:06.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:06.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:07.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.988+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:07.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.988+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:07.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.988+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:38:07.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.988+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:07.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.988+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:07.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.988+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:07.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.988+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:07.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.988+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:07.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:07.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:07.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:07.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:07.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:07.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:38:07.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:38:07.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:07.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:07.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:38:07.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:07.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:07.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015640 con 0x7f6400146050 2021-03-09T09:38:07.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:38:07.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.989+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:07.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.990+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 secure :-1 s=READY pgs=8186 cs=0 l=1 rev1=1 rx=0x7f63e000f840 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:07.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.990+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:38:07.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.990+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:07.014 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:06.990+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22321 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:07.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc032780 con 0x7f6400146050 2021-03-09T09:38:07.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.991+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:07.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.991+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:07.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.991+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:07.016 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:07.016 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:07.017 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:07.017 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:07.017 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:07.017 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:07.018 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:07.018 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:07.018 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:07.018 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:07.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032780 con 0x7f6400146050 2021-03-09T09:38:07.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.995+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:07.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.995+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:38:07.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:06.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:38:07.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:07.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:07.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:07.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:08.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:08.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:08.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:08.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:08.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:08.469+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:09.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:09.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:09.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:09.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:09.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:09.916+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:10.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:10.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:10.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:10.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:11.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:11.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:11.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:38:11.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:11.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 secure :-1 s=READY pgs=8186 cs=0 l=1 rev1=1 rx=0x7f63e000f840 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:11.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 secure :-1 s=READY pgs=8186 cs=0 l=1 rev1=1 rx=0x7f63e000f840 tx=0x7f63fc04a740).stop 2021-03-09T09:38:11.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:11.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:11.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:11.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e520 unknown :-1 s=CLOSED pgs=8186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:11.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:38:11.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:11.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:11.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:11.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:11.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:11.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:11.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:11.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:38:11.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:11.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8008840 2021-03-09T09:38:11.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:11.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266f0 con 0x7f63d8005e00 2021-03-09T09:38:12.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:12.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:12.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:12.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:12.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.991+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:12.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.992+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:12.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.992+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:38:12.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.992+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:12.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.992+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:12.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.993+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.993+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.993+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:12.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:12.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:12.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:12.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:12.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:38:12.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:38:12.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:12.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:12.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b6c0 con 0x7f63d8005e00 2021-03-09T09:38:12.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:12.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:12.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:38:12.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:38:12.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:12.008 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:11.994+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22358 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:12.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.995+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8206 cs=0 l=1 rev1=1 rx=0x7f63fc019ff0 tx=0x7f63fc0335f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:12.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.995+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:38:12.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.995+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:12.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc032780 con 0x7f63d8005e00 2021-03-09T09:38:12.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.996+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:12.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.996+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:12.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.996+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:12.011 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:12.011 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:12.011 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:12.011 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:12.012 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:12.012 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:12.012 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:12.012 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:12.013 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:12.013 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:12.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032780 con 0x7f63d8005e00 2021-03-09T09:38:12.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.996+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.996+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:38:12.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:11.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc032b70 con 0x7f63d8005e00 2021-03-09T09:38:12.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:12.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:12.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:12.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:13.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:13.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:13.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:13.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:14.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:14.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:14.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:14.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:14.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:14.917+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:15.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:15.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:15.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:15.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:16.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:16.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:16.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8206 cs=0 l=1 rev1=1 rx=0x7f63fc019ff0 tx=0x7f63fc0335f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8206 cs=0 l=1 rev1=1 rx=0x7f63fc019ff0 tx=0x7f63fc0335f0).stop 2021-03-09T09:38:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:16.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:38:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:17.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:17.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:17.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:17.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:17.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:17.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:17.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:38:17.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:17.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f4e0 con 0x7f63d8008840 2021-03-09T09:38:17.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:17.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400146050 2021-03-09T09:38:17.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:17.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:17.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:17.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:17.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.995+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:17.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.996+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:17.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.996+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:38:17.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.996+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:17.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.996+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:17.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:17.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:17.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:17.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:17.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:17.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:38:17.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:38:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b440 con 0x7f6400146050 2021-03-09T09:38:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:17.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a970 con 0x7f6400146050 2021-03-09T09:38:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:38:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:17.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:38:17.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0001030 secure :-1 s=READY pgs=8213 cs=0 l=1 rev1=1 rx=0x7f63e0010140 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:17.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:17.011 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:16.997+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22398 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:17.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:38:17.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.998+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:17.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.998+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:17.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.998+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:17.014 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:17.014 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:17.014 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:17.014 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:17.015 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:17.015 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:17.015 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:17.015 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:17.015 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:17.016 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:17.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f6400146050 2021-03-09T09:38:17.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.998+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:17.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:16.998+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:38:17.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:17.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:38:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:17.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:17.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:18.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:18.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:18.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:18.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:18.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:18.470+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:19.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:19.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:19.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:19.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:19.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:19.918+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:20.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:20.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:20.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:20.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:22.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:38:22.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:22.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0001030 secure :-1 s=READY pgs=8213 cs=0 l=1 rev1=1 rx=0x7f63e0010140 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:22.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0001030 secure :-1 s=READY pgs=8213 cs=0 l=1 rev1=1 rx=0x7f63e0010140 tx=0x7f63f004db20).stop 2021-03-09T09:38:22.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:22.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:22.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:22.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0001030 unknown :-1 s=CLOSED pgs=8213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:22.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:38:22.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:22.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:22.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:22.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:22.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:22.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:22.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:22.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065280 con 0x7f63d8003400 2021-03-09T09:38:22.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:22.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013200 con 0x7f63d8008840 2021-03-09T09:38:22.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:22.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:38:22.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:22.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:22.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:22.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:22.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.998+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:22.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.999+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:22.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.999+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:38:22.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.999+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:22.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:21.999+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:22.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:22.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:22.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:22.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:22.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:22.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:22.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:22.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:22.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:38:22.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:38:22.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:22.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:22.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ebd0 con 0x7f63d8005e00 2021-03-09T09:38:22.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:22.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:22.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b6c0 con 0x7f63d8005e00 2021-03-09T09:38:22.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:38:22.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.000+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:22.014 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:22.001+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22435 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:22.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.002+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8232 cs=0 l=1 rev1=1 rx=0x7f63fc074040 tx=0x7f63fc0777d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:22.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.002+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:38:22.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.002+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:22.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.005+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc032780 con 0x7f63d8005e00 2021-03-09T09:38:22.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.005+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:22.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.005+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:22.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.005+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:22.016 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:22.017 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:22.017 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:22.017 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:22.018 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:22.018 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:22.018 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:22.018 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:22.018 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:22.019 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:22.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63d8005e00 2021-03-09T09:38:22.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.006+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:22.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.006+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:38:22.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T09:38:22.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:22.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:22.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:23.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:23.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:23.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:23.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:24.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:24.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:24.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:24.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:24.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:24.919+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:25.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:25.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:25.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:25.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:26.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:26.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:26.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:26.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:27.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:27.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:27.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8232 cs=0 l=1 rev1=1 rx=0x7f63fc074040 tx=0x7f63fc0777d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:27.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8232 cs=0 l=1 rev1=1 rx=0x7f63fc074040 tx=0x7f63fc0777d0).stop 2021-03-09T09:38:27.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:27.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:27.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:27.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:27.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:38:27.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:27.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:27.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:27.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:27.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:27.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:27.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:27.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:38:27.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:27.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028750 con 0x7f63d8008840 2021-03-09T09:38:27.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:27.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00290e0 con 0x7f6400146050 2021-03-09T09:38:27.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:27.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:27.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:27.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:27.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.002+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:27.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.003+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:27.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.003+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:38:27.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.003+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:27.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.003+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:27.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.004+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:27.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.004+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:27.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.004+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:27.013 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22475 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:27.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:27.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:27.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:27.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:27.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:27.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:38:27.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:38:27.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:27.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:27.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:38:27.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:27.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:27.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b440 con 0x7f6400146050 2021-03-09T09:38:27.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:38:27.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.005+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:27.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.006+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:38:27.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.006+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:27.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015600 secure :-1 s=READY pgs=8250 cs=0 l=1 rev1=1 rx=0x7f63f0088e70 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:27.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f6400146050 2021-03-09T09:38:27.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.009+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:27.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.009+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:27.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.009+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:27.019 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:27.020 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:27.021 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:27.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0088ee0 con 0x7f6400146050 2021-03-09T09:38:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.012+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e290 con 0x7f6400146050 2021-03-09T09:38:27.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.012+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:38:27.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:27.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:27.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:28.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:28.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:28.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:28.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:28.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:28.471+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:29.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:29.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:29.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:29.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:29.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:29.920+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:30.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:30.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:30.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:30.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:31.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:31.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:31.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:32.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:32.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:32.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015600 secure :-1 s=READY pgs=8250 cs=0 l=1 rev1=1 rx=0x7f63f0088e70 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:32.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015600 secure :-1 s=READY pgs=8250 cs=0 l=1 rev1=1 rx=0x7f63f0088e70 tx=0x7f63f000df50).stop 2021-03-09T09:38:32.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:32.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:32.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:32.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015600 unknown :-1 s=CLOSED pgs=8250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:32.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:38:32.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:32.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:32.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:32.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:32.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:32.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:32.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:32.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:38:32.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:32.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013200 con 0x7f63d8008840 2021-03-09T09:38:32.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:32.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c590 con 0x7f63d8005e00 2021-03-09T09:38:32.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:32.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:32.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:32.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:32.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.006+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:32.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.007+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:32.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.007+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:38:32.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.007+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:32.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.007+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:32.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.007+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:32.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.007+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:32.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.007+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:32.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:32.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:32.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:32.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:32.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:32.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:38:32.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:38:32.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:32.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:32.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e750 con 0x7f63d8005e00 2021-03-09T09:38:32.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:32.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:32.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:38:32.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:38:32.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:32.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8257 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f000ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:32.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:38:32.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:32.022 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:32.008+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22512 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:32.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T09:38:32.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.009+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:32.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.009+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:32.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.009+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:32.024 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:32.024 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:32.024 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:32.025 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:32.025 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:32.025 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:32.025 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:32.025 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:32.026 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:32.026 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:32.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T09:38:32.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.010+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:32.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.010+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:38:32.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.011+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042530 con 0x7f63d8005e00 2021-03-09T09:38:32.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:32.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:32.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:33.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:33.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:33.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:33.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:34.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:34.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:34.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:34.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:34.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:34.921+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:35.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:35.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:35.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:35.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:36.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:36.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:36.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:37.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:38:37.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8257 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f000ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8257 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f000ff20).stop 2021-03-09T09:38:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:38:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:38:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015260 con 0x7f63d8008840 2021-03-09T09:38:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:37.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:38:37.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:37.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:37.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.009+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:37.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.010+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:37.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.010+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:38:37.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.010+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:37.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.010+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:37.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.010+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:37.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.010+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:37.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.010+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:37.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.010+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:37.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.010+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:37.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:37.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:37.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:37.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:37.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:37.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:38:37.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:38:37.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:37.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:37.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:38:37.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:37.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:37.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ebd0 con 0x7f6400146050 2021-03-09T09:38:37.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:38:37.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:37.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=8270 cs=0 l=1 rev1=1 rx=0x7f63fc027070 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:37.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:38:37.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.011+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:37.025 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:37.012+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22552 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:37.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc032780 con 0x7f6400146050 2021-03-09T09:38:37.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.013+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:37.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.013+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:37.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.013+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:37.027 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:37.027 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:37.027 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:37.027 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:37.028 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:37.028 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:37.028 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:37.028 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:37.028 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:37.029 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:37.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.015+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032780 con 0x7f6400146050 2021-03-09T09:38:37.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.015+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:37.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.015+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:38:37.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f6400146050 2021-03-09T09:38:37.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:37.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:37.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:38.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:38.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:38.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:38.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:38.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:38.473+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:39.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:39.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:39.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:39.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:39.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:39.922+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:40.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:40.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:40.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:40.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:41.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:41.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:41.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:41.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:42.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:42.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=8270 cs=0 l=1 rev1=1 rx=0x7f63fc027070 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=8270 cs=0 l=1 rev1=1 rx=0x7f63fc027070 tx=0x7f63fc009a00).stop 2021-03-09T09:38:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:42.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=CLOSED pgs=8270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:42.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:38:42.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:42.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:42.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:42.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:42.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:42.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:42.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:42.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013200 con 0x7f63d8003400 2021-03-09T09:38:42.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:42.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065bc0 con 0x7f63d8008840 2021-03-09T09:38:42.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:42.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015260 con 0x7f63d8005e00 2021-03-09T09:38:42.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.012+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:42.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:42.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:42.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:42.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:38:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:42.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:42.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.013+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:42.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:42.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:42.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:42.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:42.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:42.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:38:42.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:38:42.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:42.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:42.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014020 con 0x7f63d8005e00 2021-03-09T09:38:42.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:42.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:42.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:38:42.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:38:42.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:42.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8282 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc065790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:42.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:38:42.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.014+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:42.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.015+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02be60 con 0x7f63d8005e00 2021-03-09T09:38:42.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.015+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:42.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.015+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:42.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.015+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:42.029 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:42.029 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:42.029 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:42.030 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:42.030 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:42.030 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:42.031 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:42.031 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:42.031 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:42.032 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:42.033 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:42.015+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22589 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:42.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc012e60 con 0x7f63d8005e00 2021-03-09T09:38:42.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.017+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:42.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.017+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:38:42.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc032b70 con 0x7f63d8005e00 2021-03-09T09:38:42.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:42.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:42.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:43.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:43.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:43.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:43.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:44.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:44.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:44.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:44.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:44.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:44.923+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:45.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:45.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:45.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:46.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:46.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:46.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:46.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:47.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:38:47.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8282 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc065790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8282 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc065790).stop 2021-03-09T09:38:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:38:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:47.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:47.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T09:38:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00285d0 con 0x7f63d8008840 2021-03-09T09:38:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d240 con 0x7f6400146050 2021-03-09T09:38:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:47.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:47.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:47.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:47.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.016+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:47.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.017+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:47.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.017+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:38:47.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.017+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:47.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.017+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:47.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:47.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:47.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:47.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:47.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:47.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:47.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:47.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:47.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:38:47.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:38:47.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:47.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:47.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:38:47.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:47.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:47.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e750 con 0x7f6400146050 2021-03-09T09:38:47.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:38:47.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.018+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:47.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.019+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=8298 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:47.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.019+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:38:47.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.019+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:47.035 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:47.019+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22629 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:47.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f6400146050 2021-03-09T09:38:47.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.020+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:47.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.020+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:47.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.020+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:47.037 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:47.037 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:47.038 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:47.038 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:47.038 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:47.038 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:47.039 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:47.039 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:47.039 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:47.039 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:47.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f6400146050 2021-03-09T09:38:47.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.024+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:47.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.024+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:38:47.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:38:47.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:47.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:47.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:48.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:48.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:48.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:48.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:48.474+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:49.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:49.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:49.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:49.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:49.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:49.924+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:50.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:50.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:50.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:50.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:51.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:51.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:51.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:51.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:52.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:52.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=8298 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:52.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=8298 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c30).stop 2021-03-09T09:38:52.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:52.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:52.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:52.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=CLOSED pgs=8298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:52.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:38:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:52.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T09:38:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c350 con 0x7f63d8008840 2021-03-09T09:38:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:38:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.019+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.020+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:52.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.020+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:52.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.020+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:52.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.020+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:52.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:52.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:38:52.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:52.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:52.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:52.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:52.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:52.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:52.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:52.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:52.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:52.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:52.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:38:52.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:38:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e2b0 con 0x7f63d8005e00 2021-03-09T09:38:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:52.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:38:52.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:38:52.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.022+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:52.035 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:52.023+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22666 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:52.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.023+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8317 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0015990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.023+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:38:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.023+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8005e00 2021-03-09T09:38:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.023+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:52.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.023+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:52.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.023+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:52.038 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:52.038 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:52.038 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:52.039 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:52.039 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:52.040 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:52.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8005e00 2021-03-09T09:38:52.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.024+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:52.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.024+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:38:52.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029c20 con 0x7f63d8005e00 2021-03-09T09:38:52.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:52.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:52.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:53.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:53.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:53.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:53.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:54.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:54.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:54.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:54.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:54.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:54.925+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:38:55.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:55.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:55.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:55.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:56.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:56.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:56.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:57.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:38:57.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:38:57.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8317 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0015990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:38:57.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8317 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0015990).stop 2021-03-09T09:38:57.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:57.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:38:57.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:38:57.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:38:57.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:38:57.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:57.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:38:57.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:57.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:38:57.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:38:57.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:38:57.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:57.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:38:57.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:57.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f330 con 0x7f63d8008840 2021-03-09T09:38:57.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:38:57.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00285d0 con 0x7f6400146050 2021-03-09T09:38:57.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:38:57.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:38:57.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:57.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:38:57.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.024+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:38:57.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:38:57.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:38:57.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:38:57.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:38:57.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:38:57.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:38:57.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:38:57.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:38:57.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:57.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:57.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:38:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:38:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:38:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:38:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:57.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:38:57.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:38:57.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:38:57.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014020 con 0x7f6400146050 2021-03-09T09:38:57.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:38:57.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.025+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:57.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.026+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:38:57.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.026+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:38:57.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.026+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=8326 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:38:57.040 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:38:57.027+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22706 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:38:57.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T09:38:57.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.028+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:38:57.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.028+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:38:57.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.028+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:38:57.042 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:38:57.042 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:38:57.042 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:57.042 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:38:57.043 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:38:57.043 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:38:57.043 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:38:57.043 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:38:57.043 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:38:57.044 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:38:57.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T09:38:57.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.029+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:38:57.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.029+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:38:57.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.030+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:38:57.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:57.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:57.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:58.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:58.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:58.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:58.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:58.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:58.475+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:38:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:59.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:38:59.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:59.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:38:59.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:38:59.926+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:00.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:00.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:01.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:01.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:02.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:02.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:02.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=8326 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:02.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=8326 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc003a50).stop 2021-03-09T09:39:02.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:02.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:02.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:02.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=CLOSED pgs=8326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:02.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:39:02.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:02.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:02.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:02.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:02.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:02.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:02.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:02.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013ee0 con 0x7f63d8003400 2021-03-09T09:39:02.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:02.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:39:02.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:02.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8005e00 2021-03-09T09:39:02.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:02.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:02.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:02.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:02.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:39:02.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:02.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.027+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:02.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.028+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:02.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.028+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:02.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.028+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:02.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.028+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:02.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.028+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:02.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.028+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:02.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:02.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:02.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:02.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:02.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:39:02.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:39:02.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:02.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:02.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002be10 con 0x7f63d8005e00 2021-03-09T09:39:02.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:02.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:02.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e2b0 con 0x7f63d8005e00 2021-03-09T09:39:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:39:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.029+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.030+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8338 cs=0 l=1 rev1=1 rx=0x7f63f0041b20 tx=0x7f63f0029710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.030+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:39:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.030+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:02.044 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:02.030+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22743 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T09:39:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.032+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.032+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.032+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:02.049 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:02.049 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:02.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T09:39:02.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.033+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:02.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.034+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:39:02.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T09:39:02.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:02.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:02.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:03.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:03.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:03.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:03.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:04.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:04.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:04.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:04.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:04.927+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:05.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:05.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:05.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:05.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:06.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:06.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:06.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:06.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:07.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8338 cs=0 l=1 rev1=1 rx=0x7f63f0041b20 tx=0x7f63f0029710).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8338 cs=0 l=1 rev1=1 rx=0x7f63f0041b20 tx=0x7f63f0029710).stop 2021-03-09T09:39:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:07.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:39:07.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:07.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00243c0 con 0x7f63d8003400 2021-03-09T09:39:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:07.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8008840 2021-03-09T09:39:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d350 con 0x7f6400146050 2021-03-09T09:39:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:07.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:07.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.031+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:07.041 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:07.033+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22783 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:07.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.034+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:07.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.034+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:39:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.034+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.034+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:07.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:07.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:39:07.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:39:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:07.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:07.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ebd0 con 0x7f6400146050 2021-03-09T09:39:07.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:07.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:07.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002be10 con 0x7f6400146050 2021-03-09T09:39:07.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:39:07.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:07.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065360 secure :-1 s=READY pgs=8358 cs=0 l=1 rev1=1 rx=0x7f63f0038830 tx=0x7f63f00386e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:07.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:39:07.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.035+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:07.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:39:07.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.036+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:07.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.036+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:07.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.036+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:07.053 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:07.053 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:07.053 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:07.054 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:07.054 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:07.055 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:07.055 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:07.056 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:07.056 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:07.056 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:07.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:39:07.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.036+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:07.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.037+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:39:07.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.037+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f6400146050 2021-03-09T09:39:07.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:07.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:07.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:08.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:08.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:08.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:08.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:08.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:08.477+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:09.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:09.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:09.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:09.928+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:10.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:10.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:10.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:10.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:11.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:11.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:11.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:11.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:12.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0065360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:12.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0065360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:12.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065360 secure :-1 s=READY pgs=8358 cs=0 l=1 rev1=1 rx=0x7f63f0038830 tx=0x7f63f00386e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:12.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065360 secure :-1 s=READY pgs=8358 cs=0 l=1 rev1=1 rx=0x7f63f0038830 tx=0x7f63f00386e0).stop 2021-03-09T09:39:12.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:12.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:12.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0065360 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:12.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0065360 unknown :-1 s=CLOSED pgs=8358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:12.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:39:12.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.034+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:12.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:12.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:12.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:12.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:12.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:12.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:12.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:39:12.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:12.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024260 con 0x7f63d8008840 2021-03-09T09:39:12.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:12.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022a60 con 0x7f63d8005e00 2021-03-09T09:39:12.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:12.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:12.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:12.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:12.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:12.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:12.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:39:12.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:12.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.035+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:12.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.036+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:12.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.036+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:12.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.036+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:12.046 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:12.037+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22820 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:12.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:12.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:12.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:12.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:12.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:12.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:39:12.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:39:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e2b0 con 0x7f63d8005e00 2021-03-09T09:39:12.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:12.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:12.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:39:12.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:39:12.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:12.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:39:12.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8372 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc077e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:12.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.038+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:12.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:39:12.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.039+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:12.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.039+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:12.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.039+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:12.053 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:12.053 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:12.053 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:12.053 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:12.054 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:12.054 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:12.054 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:12.054 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:12.055 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:12.055 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:12.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:39:12.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.041+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:12.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.041+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:39:12.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:39:12.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:12.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:12.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:13.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:13.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:13.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:13.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:14.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:14.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:14.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:14.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:14.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:14.929+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:15.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:15.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:15.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:15.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:16.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:16.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:16.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:16.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:17.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:17.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:17.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8372 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc077e10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:17.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8372 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc077e10).stop 2021-03-09T09:39:17.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:17.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:17.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:17.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:17.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:39:17.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:17.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:17.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:17.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:17.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:17.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:17.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:17.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8003400 2021-03-09T09:39:17.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:17.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:39:17.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:17.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002abc0 con 0x7f6400146050 2021-03-09T09:39:17.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:17.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:17.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:17.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:17.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.038+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:17.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.039+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:17.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.039+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:39:17.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.039+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:17.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.039+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:17.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.039+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:17.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.039+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:17.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.039+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:17.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:17.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:17.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:17.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:39:17.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:39:17.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:17.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:17.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:39:17.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:17.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:17.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e2b0 con 0x7f6400146050 2021-03-09T09:39:17.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:39:17.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.040+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:17.056 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:17.041+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22860 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:17.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.041+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:39:17.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.041+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:17.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.041+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8386 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc0219e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:17.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T09:39:17.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.042+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:17.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.042+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:17.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.042+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:17.059 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:17.059 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:17.060 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:17.060 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:17.060 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:17.060 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:17.060 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:17.061 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:17.061 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:17.061 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:17.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T09:39:17.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.042+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:17.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.043+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:39:17.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021460 con 0x7f6400146050 2021-03-09T09:39:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:17.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:18.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:18.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:18.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:18.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:18.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:18.477+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:19.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:19.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:19.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:19.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:19.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:19.930+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:20.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:20.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:20.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:20.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:21.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:21.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:21.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:21.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:22.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:22.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:22.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8386 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc0219e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:22.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8386 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc0219e0).stop 2021-03-09T09:39:22.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:22.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:22.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:22.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=CLOSED pgs=8386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:22.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:39:22.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:22.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:22.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:22.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:22.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:22.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:22.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:22.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d440 con 0x7f63d8003400 2021-03-09T09:39:22.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:22.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:39:22.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:22.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cc40 con 0x7f63d8005e00 2021-03-09T09:39:22.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:22.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:22.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:22.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:22.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:22.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:22.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:39:22.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:22.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.042+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:22.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.043+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:22.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.043+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:22.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.043+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:22.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:22.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:22.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:22.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:22.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:22.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:39:22.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:39:22.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:22.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:22.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:39:22.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:22.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:22.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001e40 con 0x7f63d8005e00 2021-03-09T09:39:22.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:39:22.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:22.057 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22895 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:22.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:39:22.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8393 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4003b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:22.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.044+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:22.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.046+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40225f0 con 0x7f63d8005e00 2021-03-09T09:39:22.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.046+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:22.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.046+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:22.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.046+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:22.060 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:22.060 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:22.060 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:22.061 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:22.061 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:22.061 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:22.061 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:22.061 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:22.062 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:22.062 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:22.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40225f0 con 0x7f63d8005e00 2021-03-09T09:39:22.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.048+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:22.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.048+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:39:22.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63d8005e00 2021-03-09T09:39:22.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:22.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:22.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:23.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:23.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:23.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:23.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:24.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:24.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:24.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:24.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:24.931+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:25.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:25.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:25.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:25.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:26.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:26.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:26.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:26.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:27.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:27.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:27.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8393 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4003b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:27.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8393 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4003b40).stop 2021-03-09T09:39:27.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:27.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:27.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:27.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:27.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:39:27.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:27.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:27.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:27.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:27.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:27.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:27.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:27.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8003400 2021-03-09T09:39:27.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:27.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fe00 con 0x7f63d8008840 2021-03-09T09:39:27.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:27.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f200 con 0x7f6400146050 2021-03-09T09:39:27.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:27.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:27.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.045+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:27.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.046+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:27.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.046+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:27.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.047+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:27.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.047+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:39:27.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.047+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:27.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.047+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:27.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.048+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:27.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.048+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:27.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.048+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:27.061 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:27.048+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22937 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:27.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:27.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:27.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:27.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:27.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:27.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:39:27.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:39:27.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:27.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:27.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f6400146050 2021-03-09T09:39:27.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:27.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:27.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ebd0 con 0x7f6400146050 2021-03-09T09:39:27.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:39:27.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:27.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8415 cs=0 l=1 rev1=1 rx=0x7f63e0013570 tx=0x7f63f0029790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:27.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:39:27.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:27.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f6400146050 2021-03-09T09:39:27.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:27.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:27.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.051+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:27.067 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:27.068 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:27.068 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:27.068 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:27.068 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:27.068 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:27.069 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:27.069 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:27.069 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:27.069 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:27.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f6400146050 2021-03-09T09:39:27.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.054+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:27.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f6400146050 2021-03-09T09:39:27.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.054+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:39:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:27.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:28.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:28.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:28.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:28.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:28.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:28.479+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:29.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:29.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:29.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:29.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:29.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:29.932+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:30.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:30.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:30.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:30.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:31.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:31.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:31.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:31.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:32.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:32.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:32.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8415 cs=0 l=1 rev1=1 rx=0x7f63e0013570 tx=0x7f63f0029790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:32.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8415 cs=0 l=1 rev1=1 rx=0x7f63e0013570 tx=0x7f63f0029790).stop 2021-03-09T09:39:32.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:32.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:32.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:32.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=CLOSED pgs=8415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:32.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:39:32.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:32.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:32.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:32.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:32.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:32.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:32.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:32.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8003400 2021-03-09T09:39:32.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:32.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:39:32.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:32.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b380 con 0x7f63d8005e00 2021-03-09T09:39:32.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:32.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:32.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.049+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:32.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:32.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.050+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:39:32.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.050+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:32.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.050+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:32.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.050+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:32.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.050+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:32.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.050+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:32.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:32.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:32.061 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:32.052+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 22974 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:32.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:32.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:32.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:32.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:32.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:32.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:39:32.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:39:32.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:32.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:32.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ebd0 con 0x7f63d8005e00 2021-03-09T09:39:32.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:32.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:32.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63d8005e00 2021-03-09T09:39:32.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:39:32.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:32.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8429 cs=0 l=1 rev1=1 rx=0x7f63f0088e70 tx=0x7f63f0006690).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:32.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:39:32.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.053+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:32.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0088cb0 con 0x7f63d8005e00 2021-03-09T09:39:32.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.055+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:32.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.055+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:32.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.055+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:32.067 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:32.067 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:32.068 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:32.068 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:32.068 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:32.068 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:32.069 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:32.069 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:32.069 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:32.069 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:32.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:39:32.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.057+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:32.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.057+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:39:32.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038a20 con 0x7f63d8005e00 2021-03-09T09:39:32.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:32.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:32.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:33.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:33.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:33.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:33.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:34.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:34.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:34.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:34.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:34.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:34.933+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:35.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:35.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:35.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:35.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:36.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:36.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:36.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:36.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:37.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:37.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:37.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8429 cs=0 l=1 rev1=1 rx=0x7f63f0088e70 tx=0x7f63f0006690).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:37.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8429 cs=0 l=1 rev1=1 rx=0x7f63f0088e70 tx=0x7f63f0006690).stop 2021-03-09T09:39:37.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:37.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:37.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:37.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:37.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:39:37.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:37.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:37.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:37.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:37.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:37.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:37.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:37.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:39:37.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:37.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fe00 con 0x7f63d8008840 2021-03-09T09:39:37.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:37.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dd60 con 0x7f6400146050 2021-03-09T09:39:37.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:37.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:37.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:37.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:37.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.053+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:37.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.054+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:37.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.054+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:39:37.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.054+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:37.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.054+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:37.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.054+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:37.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.054+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:37.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.054+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:37.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:37.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:37.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:37.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:37.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:37.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:39:37.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:39:37.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:39:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:37.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:37.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:39:37.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:39:37.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.055+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:37.067 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:37.056+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23014 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:37.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8442 cs=0 l=1 rev1=1 rx=0x7f63e002e280 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:37.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.059+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:39:37.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.059+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:37.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.061+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T09:39:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.061+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.061+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.061+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:37.070 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:37.071 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:37.071 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:37.071 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:37.071 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:37.072 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:37.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T09:39:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.063+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.063+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:39:37.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021460 con 0x7f6400146050 2021-03-09T09:39:37.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:37.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:37.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:38.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:38.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:38.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:38.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:38.480+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:39.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:39.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:39.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:39.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:39.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:39.934+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:40.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:40.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:40.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:40.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:41.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:41.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:41.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:41.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:42.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:42.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:42.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8442 cs=0 l=1 rev1=1 rx=0x7f63e002e280 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:42.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8442 cs=0 l=1 rev1=1 rx=0x7f63e002e280 tx=0x7f63fc009a00).stop 2021-03-09T09:39:42.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:42.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:42.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:42.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=CLOSED pgs=8442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:42.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:39:42.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:42.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:42.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:42.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:42.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:42.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:42.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:42.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:39:42.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:42.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T09:39:42.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:42.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8005e00 2021-03-09T09:39:42.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:42.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:42.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:42.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:42.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:42.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:42.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:39:42.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:42.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.057+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:42.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.058+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:42.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.058+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:42.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.058+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:42.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:42.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:42.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:42.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:42.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:42.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:39:42.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:39:42.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:42.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:42.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012d40 con 0x7f63d8005e00 2021-03-09T09:39:42.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:42.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:42.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ebd0 con 0x7f63d8005e00 2021-03-09T09:39:42.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:39:42.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:42.071 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23051 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:42.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:39:42.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:42.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8452 cs=0 l=1 rev1=1 rx=0x7f63f002f040 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:42.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028ec0 con 0x7f63d8005e00 2021-03-09T09:39:42.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.060+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:42.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.060+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:42.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.060+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:42.074 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:42.075 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:42.075 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:42.075 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:42.075 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:42.075 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:42.076 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:42.076 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:42.076 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:42.076 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:42.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028ec0 con 0x7f63d8005e00 2021-03-09T09:39:42.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.066+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:42.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.066+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:39:42.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63d8005e00 2021-03-09T09:39:42.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:42.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:42.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:43.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:43.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:43.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:43.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:44.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:44.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:44.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:44.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:44.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:44.935+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:45.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:45.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:45.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:45.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:46.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:46.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:46.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:46.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:47.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:47.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:47.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8452 cs=0 l=1 rev1=1 rx=0x7f63f002f040 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:47.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8452 cs=0 l=1 rev1=1 rx=0x7f63f002f040 tx=0x7f63f0000c30).stop 2021-03-09T09:39:47.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:47.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:47.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:47.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:47.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:39:47.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:47.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:47.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:47.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:47.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:47.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:47.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:47.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dd60 con 0x7f63d8003400 2021-03-09T09:39:47.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:47.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:39:47.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:47.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fe00 con 0x7f6400146050 2021-03-09T09:39:47.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:47.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:47.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:47.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:47.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.060+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:47.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:47.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.061+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:39:47.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.061+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:47.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.061+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:47.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:47.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:47.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:47.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:47.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:47.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:47.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:47.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:39:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:39:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:47.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f6400146050 2021-03-09T09:39:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:47.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:39:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:39:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.062+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.063+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:39:47.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.063+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8467 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:47.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.063+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:47.077 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:47.063+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23091 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:47.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02ba00 con 0x7f6400146050 2021-03-09T09:39:47.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.065+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:47.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.065+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:47.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.065+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:47.079 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:47.079 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:47.080 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:47.081 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02ba00 con 0x7f6400146050 2021-03-09T09:39:47.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.068+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.068+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:39:47.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f6400146050 2021-03-09T09:39:47.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:47.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:47.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:48.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:48.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:48.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:48.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:48.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:48.482+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:49.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:49.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:49.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:49.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:49.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:49.936+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:50.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:50.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:50.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:50.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:51.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:51.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:51.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:51.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:52.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:52.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:52.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8467 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc014000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:52.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 secure :-1 s=READY pgs=8467 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc014000).stop 2021-03-09T09:39:52.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:52.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:52.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b1e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:52.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b1e0 unknown :-1 s=CLOSED pgs=8467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:52.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:39:52.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:52.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:52.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:52.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:52.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:52.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:52.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:52.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00303c0 con 0x7f63d8003400 2021-03-09T09:39:52.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:52.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8008840 2021-03-09T09:39:52.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:52.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:39:52.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:52.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:52.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.064+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.065+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.065+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:52.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.065+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:52.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.065+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:39:52.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.065+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:52.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.065+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:52.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.065+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:52.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.065+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:52.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.065+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:52.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:52.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:52.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:39:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:39:52.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016ed0 con 0x7f63d8005e00 2021-03-09T09:39:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:52.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012d40 con 0x7f63d8005e00 2021-03-09T09:39:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:39:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:52.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:39:52.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8477 cs=0 l=1 rev1=1 rx=0x7f63e004e0a0 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:52.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.066+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:52.096 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:52.067+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23128 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:52.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T09:39:52.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.067+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:52.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.067+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:52.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.067+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:52.098 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:52.099 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:52.099 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:52.099 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:52.100 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:52.100 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:52.100 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:52.100 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:52.101 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:52.101 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:52.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T09:39:52.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.069+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:52.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.069+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:39:52.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T09:39:52.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:52.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:52.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:53.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:53.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:53.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:53.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:54.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:54.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:54.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:54.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:54.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:54.937+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:39:55.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:55.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:55.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:55.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:56.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:56.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:56.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:56.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:57.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:39:57.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:39:57.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8477 cs=0 l=1 rev1=1 rx=0x7f63e004e0a0 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:39:57.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8477 cs=0 l=1 rev1=1 rx=0x7f63e004e0a0 tx=0x7f63f0013b80).stop 2021-03-09T09:39:57.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:57.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:39:57.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:39:57.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:39:57.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:39:57.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:57.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:39:57.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:57.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:39:57.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:39:57.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:39:57.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:57.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026010 con 0x7f63d8003400 2021-03-09T09:39:57.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:57.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8008840 2021-03-09T09:39:57.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:39:57.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010680 con 0x7f6400146050 2021-03-09T09:39:57.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:39:57.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:39:57.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:57.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:39:57.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:39:57.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:39:57.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:39:57.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:39:57.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:39:57.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:39:57.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:39:57.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.068+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:39:57.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:39:57.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:57.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:57.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:39:57.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:39:57.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:39:57.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:39:57.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:57.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:57.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c8b0 con 0x7f6400146050 2021-03-09T09:39:57.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:39:57.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:39:57.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f6400146050 2021-03-09T09:39:57.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:39:57.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:57.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ba90 secure :-1 s=READY pgs=8489 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc02bae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:39:57.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:39:57.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.069+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:39:57.086 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:39:57.070+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23168 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:39:57.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f6400146050 2021-03-09T09:39:57.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.071+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:39:57.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.071+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:39:57.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.071+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:39:57.088 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:39:57.088 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:39:57.088 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:57.089 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:39:57.089 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:39:57.089 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:39:57.089 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:39:57.089 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:39:57.090 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:39:57.090 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:39:57.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:39:57.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.071+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:39:57.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.071+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:39:57.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f6400146050 2021-03-09T09:39:57.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:57.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:57.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:58.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:58.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:58.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:58.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:58.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:58.483+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:39:59.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:59.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:39:59.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:59.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:39:59.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:39:59.938+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:00.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:00.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:00.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:00.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:01.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:01.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ba90 secure :-1 s=READY pgs=8489 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc02bae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:02.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ba90 secure :-1 s=READY pgs=8489 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc02bae0).stop 2021-03-09T09:40:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001ba90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:02.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001ba90 unknown :-1 s=CLOSED pgs=8489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:40:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:02.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:02.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:02.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:02.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:02.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:02.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dd60 con 0x7f63d8003400 2021-03-09T09:40:02.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:02.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:40:02.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:02.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030060 con 0x7f63d8005e00 2021-03-09T09:40:02.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:02.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:02.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:02.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:02.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.071+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:02.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.072+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:02.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.072+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:40:02.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.072+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:02.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.072+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:02.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:02.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:02.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:02.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:02.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:02.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:02.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:02.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:02.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:40:02.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:40:02.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:02.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:02.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ebd0 con 0x7f63d8005e00 2021-03-09T09:40:02.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:02.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:02.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:40:02.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:40:02.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.073+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:02.092 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:02.074+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23205 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:02.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.074+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8505 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40051f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:02.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.075+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:40:02.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.075+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:02.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63d8005e00 2021-03-09T09:40:02.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.075+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:02.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.075+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:02.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.075+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:02.095 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:02.095 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:02.095 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:02.095 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:02.096 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:02.096 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:02.096 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:02.096 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:02.096 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:02.097 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:02.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f63d8005e00 2021-03-09T09:40:02.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.076+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:02.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.076+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:40:02.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.077+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63d8005e00 2021-03-09T09:40:02.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:02.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:02.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:03.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:03.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:03.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:03.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:04.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:04.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:04.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:04.939+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:05.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:05.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:05.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:05.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:06.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:06.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:06.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:06.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:07.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:07.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:07.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8505 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40051f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:07.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8505 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40051f0).stop 2021-03-09T09:40:07.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:07.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:07.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:07.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:07.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:40:07.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:07.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:07.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:07.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:07.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:07.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:07.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:07.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8003400 2021-03-09T09:40:07.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:07.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8008840 2021-03-09T09:40:07.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:07.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fa30 con 0x7f6400146050 2021-03-09T09:40:07.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:07.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:07.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:07.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:07.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:07.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:07.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:40:07.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:07.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:07.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:07.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:07.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.075+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:07.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:07.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:07.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:07.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:07.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:07.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:40:07.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:40:07.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:07.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:07.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:40:07.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:07.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:07.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c8b0 con 0x7f6400146050 2021-03-09T09:40:07.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:40:07.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.076+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:07.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.077+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8515 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc04a7c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:07.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.077+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:40:07.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.077+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:07.092 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:07.078+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23245 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:07.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f6400146050 2021-03-09T09:40:07.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.080+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:07.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.080+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:07.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.080+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:07.093 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:07.093 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:07.094 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:07.094 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:07.094 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:07.094 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:07.095 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:07.095 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:07.095 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:07.095 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:07.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02ceb0 con 0x7f6400146050 2021-03-09T09:40:07.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.080+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:07.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02bba0 con 0x7f6400146050 2021-03-09T09:40:07.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.080+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:40:07.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:07.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:07.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:08.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:08.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:08.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:08.484+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:09.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:09.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:09.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:09.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:09.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:09.940+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:10.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:10.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:10.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:10.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:11.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:11.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:11.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:12.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:12.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8515 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc04a7c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8515 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc04a7c0).stop 2021-03-09T09:40:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:12.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:12.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:12.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=CLOSED pgs=8515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:12.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:40:12.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:12.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:12.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:12.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c770 con 0x7f63d8003400 2021-03-09T09:40:12.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:12.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064d10 con 0x7f63d8008840 2021-03-09T09:40:12.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:12.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dd60 con 0x7f63d8005e00 2021-03-09T09:40:12.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:12.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:12.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:12.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:12.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.079+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.081+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.081+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:40:12.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.081+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:12.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.081+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:12.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.081+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:12.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.081+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:12.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.081+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:12.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:12.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:12.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:12.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:12.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:12.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:40:12.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:40:12.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:12.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:12.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c8b0 con 0x7f63d8005e00 2021-03-09T09:40:12.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:12.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:12.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ebd0 con 0x7f63d8005e00 2021-03-09T09:40:12.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:40:12.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.082+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:12.093 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:12.083+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23280 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:12.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.083+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8536 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4061a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:12.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.083+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:40:12.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.083+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:12.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.084+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4061cd0 con 0x7f63d8005e00 2021-03-09T09:40:12.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.084+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:12.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.084+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:12.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.084+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:12.096 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:12.096 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:12.096 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:12.096 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:12.096 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:12.097 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:12.097 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:12.097 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:12.097 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:12.097 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:12.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.085+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022540 con 0x7f63d8005e00 2021-03-09T09:40:12.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.085+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:12.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.085+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:40:12.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40224c0 con 0x7f63d8005e00 2021-03-09T09:40:12.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:12.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:12.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:13.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:13.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:13.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:13.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:14.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:14.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:14.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:14.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:14.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:14.941+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:15.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:15.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:15.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:15.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:16.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:16.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:16.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:16.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:17.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:17.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:17.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8536 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4061a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:17.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8536 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4061a50).stop 2021-03-09T09:40:17.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:17.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:17.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:17.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:17.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:40:17.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:17.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:17.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:17.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:17.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:17.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:17.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:17.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:40:17.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:17.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a450 con 0x7f63d8008840 2021-03-09T09:40:17.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:17.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:40:17.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:17.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.083+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:17.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:40:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:17.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:17.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:17.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:40:17.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:40:17.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:17.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:17.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0023010 con 0x7f6400146050 2021-03-09T09:40:17.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:40:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:40:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.084+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:17.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.085+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:40:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.085+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:17.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.085+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8543 cs=0 l=1 rev1=1 rx=0x7f63e0024020 tx=0x7f63fc012e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:17.097 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:17.086+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23322 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:17.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03dd60 con 0x7f6400146050 2021-03-09T09:40:17.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.087+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:17.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.087+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:17.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.087+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:17.099 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:17.100 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:17.100 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:17.100 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:17.100 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:17.101 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:17.101 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:17.101 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:17.101 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:17.101 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:17.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03dd60 con 0x7f6400146050 2021-03-09T09:40:17.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.087+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:17.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.087+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:40:17.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:40:17.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:17.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:17.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:18.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:18.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:18.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:18.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:18.486+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:19.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:19.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:19.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:19.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:19.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:19.942+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:20.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:20.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:20.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:20.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:21.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:21.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:21.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:22.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8543 cs=0 l=1 rev1=1 rx=0x7f63e0024020 tx=0x7f63fc012e40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8543 cs=0 l=1 rev1=1 rx=0x7f63e0024020 tx=0x7f63fc012e40).stop 2021-03-09T09:40:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=CLOSED pgs=8543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:22.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:40:22.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:22.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:22.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:22.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8003400 2021-03-09T09:40:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c770 con 0x7f63d8008840 2021-03-09T09:40:22.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8005e00 2021-03-09T09:40:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:22.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:22.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.087+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:22.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.088+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:22.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.088+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:40:22.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.088+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:22.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.088+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:22.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.088+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:22.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.088+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:22.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.088+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:22.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:22.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:22.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:22.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:22.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:22.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:40:22.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:40:22.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:22.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:22.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:40:22.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:22.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:22.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.089+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0023010 con 0x7f63d8005e00 2021-03-09T09:40:22.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.090+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:40:22.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.090+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:22.101 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:22.090+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23359 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:22.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.090+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8562 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:22.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.091+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:40:22.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.091+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:22.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.091+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc033990 con 0x7f63d8005e00 2021-03-09T09:40:22.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.091+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:22.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.091+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:22.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.091+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:22.103 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:22.104 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:22.104 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:22.104 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:22.104 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:22.104 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:22.105 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:22.105 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:22.105 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:22.105 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:22.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.092+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc033990 con 0x7f63d8005e00 2021-03-09T09:40:22.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.092+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:22.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.092+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:40:22.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc033c20 con 0x7f63d8005e00 2021-03-09T09:40:22.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:22.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:22.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:23.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:23.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:23.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:23.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:24.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:24.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:24.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:24.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:24.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:24.943+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:25.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:25.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:25.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:25.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:26.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:26.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:26.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:26.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:27.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8562 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a750).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:27.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8562 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a750).stop 2021-03-09T09:40:27.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:27.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:40:27.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:27.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:27.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:27.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:27.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:27.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:27.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T09:40:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:40:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:40:27.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.090+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:27.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:27.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:27.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:27.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:27.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:40:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:27.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:27.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:27.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.091+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:27.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:27.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:27.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:27.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:27.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:27.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:40:27.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:40:27.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:27.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:27.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018fd0 con 0x7f6400146050 2021-03-09T09:40:27.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:27.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:27.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c8b0 con 0x7f6400146050 2021-03-09T09:40:27.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:40:27.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:27.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 secure :-1 s=READY pgs=8572 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:27.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:40:27.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.092+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:27.118 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:27.094+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23399 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:27.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4046700 con 0x7f6400146050 2021-03-09T09:40:27.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.094+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:27.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.094+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:27.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.094+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:27.120 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:27.121 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:27.121 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:27.122 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:27.122 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:27.123 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:27.123 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:27.123 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:27.124 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:27.124 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:27.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40468a0 con 0x7f6400146050 2021-03-09T09:40:27.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.094+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:27.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.094+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:40:27.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4046a70 con 0x7f6400146050 2021-03-09T09:40:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:27.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:28.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:28.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:28.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:28.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:28.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:28.487+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:29.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:29.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:29.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:29.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:29.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:29.944+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:30.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:30.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:30.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:30.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:31.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:31.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:31.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:31.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:32.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:32.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:32.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 secure :-1 s=READY pgs=8572 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:32.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 secure :-1 s=READY pgs=8572 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003b40).stop 2021-03-09T09:40:32.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:32.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:32.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:32.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=8572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:32.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:40:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:32.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:32.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:32.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:32.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dd60 con 0x7f63d8003400 2021-03-09T09:40:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8008840 2021-03-09T09:40:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:40:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:32.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:32.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:32.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:32.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:40:32.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:32.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:32.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:32.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:32.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:32.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:32.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.094+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:32.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:32.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:32.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:32.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:32.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:32.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:40:32.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:40:32.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:32.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:32.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e750 con 0x7f63d8005e00 2021-03-09T09:40:32.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:32.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:32.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018fd0 con 0x7f63d8005e00 2021-03-09T09:40:32.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:40:32.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.095+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:32.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.096+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:40:32.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.096+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:32.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.096+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8586 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f401aef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:32.108 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:32.097+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23436 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:32.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400ee80 con 0x7f63d8005e00 2021-03-09T09:40:32.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.098+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:32.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.098+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:32.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.098+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:32.110 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:32.110 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:32.110 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:32.110 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:32.111 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:32.111 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:32.111 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:32.111 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:32.112 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:32.112 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:32.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63d8005e00 2021-03-09T09:40:32.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.098+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:32.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.098+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:40:32.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.099+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005b70 con 0x7f63d8005e00 2021-03-09T09:40:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:32.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:33.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:33.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:33.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:33.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:34.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:34.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:34.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:34.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:34.945+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:35.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:35.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:35.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:35.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:36.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:36.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:36.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:36.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:37.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.096+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.096+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.096+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8586 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f401aef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8586 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f401aef0).stop 2021-03-09T09:40:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:40:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:37.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:37.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:37.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:37.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00151e0 con 0x7f63d8003400 2021-03-09T09:40:37.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026010 con 0x7f63d8008840 2021-03-09T09:40:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f6400146050 2021-03-09T09:40:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.097+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.098+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.098+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:40:37.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.098+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:37.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.098+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:37.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.098+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:37.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.098+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:37.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.098+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:37.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:37.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:37.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:37.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:37.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:37.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:40:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:40:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010430 con 0x7f6400146050 2021-03-09T09:40:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:37.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:37.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e750 con 0x7f6400146050 2021-03-09T09:40:37.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:40:37.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.099+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:37.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.100+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8605 cs=0 l=1 rev1=1 rx=0x7f63f401fb40 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:37.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.100+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:40:37.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.100+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:37.113 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:37.100+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23476 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:37.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401f980 con 0x7f6400146050 2021-03-09T09:40:37.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.101+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:37.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.101+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.101+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:37.116 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:37.116 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:37.116 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:37.116 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:37.116 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:37.117 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:37.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f401f980 con 0x7f6400146050 2021-03-09T09:40:37.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.102+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:37.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.102+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:40:37.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40467b0 con 0x7f6400146050 2021-03-09T09:40:37.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:37.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:37.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:38.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:38.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:38.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:38.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:38.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:38.487+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:39.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:39.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:39.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:39.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:39.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:39.945+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:40.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:40.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:40.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:40.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:41.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:41.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:41.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:41.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:42.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:42.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:42.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8605 cs=0 l=1 rev1=1 rx=0x7f63f401fb40 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:42.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8605 cs=0 l=1 rev1=1 rx=0x7f63f401fb40 tx=0x7f63f4017ed0).stop 2021-03-09T09:40:42.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:42.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:42.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:42.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=CLOSED pgs=8605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:42.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:40:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T09:40:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de80 con 0x7f63d8008840 2021-03-09T09:40:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T09:40:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:42.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:42.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.102+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:42.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.102+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:42.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.102+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:40:42.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.102+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:42.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.102+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:42.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.102+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:42.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.102+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:42.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.102+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:42.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:42.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:42.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:42.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:42.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:42.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:40:42.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:40:42.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:42.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:42.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d30 con 0x7f63d8005e00 2021-03-09T09:40:42.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:42.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:40:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:40:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.103+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.104+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8616 cs=0 l=1 rev1=1 rx=0x7f63fc04a830 tx=0x7f63fc012eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.104+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:40:42.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.104+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:42.117 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:42.104+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23513 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:42.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc033520 con 0x7f63d8005e00 2021-03-09T09:40:42.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.105+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:42.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.105+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:42.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.105+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:42.119 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:42.119 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:42.119 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:42.119 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:42.119 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:42.120 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:42.120 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:42.120 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:42.120 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:42.120 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:42.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:40:42.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.105+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:42.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.105+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:40:42.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63d8005e00 2021-03-09T09:40:42.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:42.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:42.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:43.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:43.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:43.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:43.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:44.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:44.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:44.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:44.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:44.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:44.946+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:45.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:45.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:45.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:45.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:46.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:46.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:46.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:46.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:47.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:47.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:47.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8616 cs=0 l=1 rev1=1 rx=0x7f63fc04a830 tx=0x7f63fc012eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:47.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8616 cs=0 l=1 rev1=1 rx=0x7f63fc04a830 tx=0x7f63fc012eb0).stop 2021-03-09T09:40:47.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:47.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:47.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:47.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:47.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:40:47.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:47.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:47.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:47.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:47.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:47.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:47.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:47.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8003400 2021-03-09T09:40:47.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:47.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T09:40:47.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:47.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012550 con 0x7f6400146050 2021-03-09T09:40:47.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.104+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:47.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.105+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:47.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.105+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:47.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.105+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:47.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.105+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:47.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.105+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:47.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.106+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:40:47.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.106+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:47.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.106+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:47.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.106+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:47.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.106+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:47.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.106+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:47.116 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23553 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:47.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:47.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:47.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:47.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:47.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:47.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:40:47.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:40:47.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:47.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:47.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015050 con 0x7f6400146050 2021-03-09T09:40:47.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:47.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016ed0 con 0x7f6400146050 2021-03-09T09:40:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:40:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.108+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8633 cs=0 l=1 rev1=1 rx=0x7f63e00095c0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.109+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:40:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.109+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:40:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.110+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.110+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.110+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:47.122 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:47.122 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:47.122 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:47.122 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:47.124 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:47.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:40:47.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.112+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:47.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.112+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:40:47.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.114+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:40:47.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:47.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:47.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:48.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:48.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:48.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:48.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:48.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:48.489+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:49.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:49.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:49.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:49.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:49.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:49.947+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:50.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:50.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:50.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:50.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:51.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:51.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:51.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:51.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:52.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:40:52.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:52.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8633 cs=0 l=1 rev1=1 rx=0x7f63e00095c0 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 secure :-1 s=READY pgs=8633 cs=0 l=1 rev1=1 rx=0x7f63e00095c0 tx=0x7f63f004db20).stop 2021-03-09T09:40:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0068400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:52.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0068400 unknown :-1 s=CLOSED pgs=8633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:52.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:40:52.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:52.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:52.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:52.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:52.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:52.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:52.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:52.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T09:40:52.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:52.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d550 con 0x7f63d8008840 2021-03-09T09:40:52.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:52.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00151e0 con 0x7f63d8005e00 2021-03-09T09:40:52.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:52.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:52.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.108+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:52.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.109+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:52.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.109+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:52.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.109+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:52.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.109+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:40:52.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.109+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:52.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.109+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:52.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.109+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:52.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.109+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:52.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.109+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:52.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:52.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:52.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:52.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:52.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:52.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:40:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:40:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:40:52.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015050 con 0x7f63d8005e00 2021-03-09T09:40:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:40:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.110+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.111+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8641 cs=0 l=1 rev1=1 rx=0x7f63f0048070 tx=0x7f63f005ff70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.111+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:40:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.111+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:52.124 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:52.111+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23590 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.113+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007fed0 con 0x7f63d8005e00 2021-03-09T09:40:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.113+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.113+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.113+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:52.126 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:52.126 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:52.126 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:52.126 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:52.128 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:52.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.115+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007fed0 con 0x7f63d8005e00 2021-03-09T09:40:52.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.115+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:52.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.115+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:40:52.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00368d0 con 0x7f63d8005e00 2021-03-09T09:40:52.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:52.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:52.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:53.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:53.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:53.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:53.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:54.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:54.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:54.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:54.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:54.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:54.948+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:40:55.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:55.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:55.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:55.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:56.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:56.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:56.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:56.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:57.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:40:57.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:40:57.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8641 cs=0 l=1 rev1=1 rx=0x7f63f0048070 tx=0x7f63f005ff70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:40:57.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8641 cs=0 l=1 rev1=1 rx=0x7f63f0048070 tx=0x7f63f005ff70).stop 2021-03-09T09:40:57.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:57.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:40:57.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:40:57.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:40:57.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:40:57.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:57.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:40:57.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0076f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:57.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:40:57.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:40:57.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:40:57.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:57.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:40:57.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:57.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8008840 2021-03-09T09:40:57.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:40:57.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ca30 con 0x7f6400146050 2021-03-09T09:40:57.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:40:57.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:40:57.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:57.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:40:57.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:40:57.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.113+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0076f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:40:57.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.113+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:40:57.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.113+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:40:57.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.113+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:40:57.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.113+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:40:57.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.113+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:40:57.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.113+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:40:57.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:40:57.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:57.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:57.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:40:57.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:40:57.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:40:57.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:40:57.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:57.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:57.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00018d0 con 0x7f6400146050 2021-03-09T09:40:57.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:40:57.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:40:57.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:40:57.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:40:57.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:57.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:40:57.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0076f60 secure :-1 s=READY pgs=8656 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc0039e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:40:57.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.114+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:40:57.129 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:40:57.115+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23630 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:40:57.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065e60 con 0x7f6400146050 2021-03-09T09:40:57.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.116+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:40:57.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.116+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:40:57.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.116+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:40:57.131 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:40:57.131 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:40:57.131 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:57.131 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:40:57.131 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:40:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:40:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:40:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:40:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:40:57.132 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:40:57.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:40:57.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.116+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:40:57.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.117+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:40:57.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:40:57.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:57.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:57.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:58.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:58.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:58.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:58.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:58.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:58.490+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:40:59.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:59.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:40:59.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:59.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:40:59.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:40:59.949+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:00.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:00.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:00.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:00.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:01.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:01.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:01.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:02.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0076f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0076f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0076f60 secure :-1 s=READY pgs=8656 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc0039e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0076f60 secure :-1 s=READY pgs=8656 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc0039e0).stop 2021-03-09T09:41:02.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0076f60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0076f60 unknown :-1 s=CLOSED pgs=8656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:02.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:41:02.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.115+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:02.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:02.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:02.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:02.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:02.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:02.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:02.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T09:41:02.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8008840 2021-03-09T09:41:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ae30 con 0x7f63d8005e00 2021-03-09T09:41:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:02.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:02.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:02.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:02.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.116+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:02.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:02.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:41:02.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:02.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:02.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:02.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:02.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:02.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:02.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:41:02.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:41:02.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.117+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:02.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:02.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016ed0 con 0x7f63d8005e00 2021-03-09T09:41:02.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:02.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:02.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00018d0 con 0x7f63d8005e00 2021-03-09T09:41:02.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:41:02.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:02.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8669 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc023b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:02.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:41:02.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.118+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:02.131 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:02.119+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23672 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:02.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f63d8005e00 2021-03-09T09:41:02.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.119+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:02.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.119+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:02.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.119+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:02.133 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:02.133 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:02.133 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:02.133 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:02.133 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:02.134 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:02.134 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:02.134 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:02.134 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:02.134 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f63d8005e00 2021-03-09T09:41:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.119+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.119+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:41:02.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.120+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63d8005e00 2021-03-09T09:41:02.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:02.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:02.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:03.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:03.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:03.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:03.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:04.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:04.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:04.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:04.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:04.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:04.950+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:05.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:05.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:05.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:05.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:06.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:06.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:06.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:07.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:07.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:07.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8669 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc023b00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:07.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8669 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc023b00).stop 2021-03-09T09:41:07.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:07.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:07.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:07.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:07.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:41:07.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:07.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:07.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:07.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:07.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:07.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:07.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:07.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026200 con 0x7f63d8003400 2021-03-09T09:41:07.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:07.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8008840 2021-03-09T09:41:07.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:07.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ca30 con 0x7f6400146050 2021-03-09T09:41:07.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.119+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:07.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:07.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:07.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:07.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:07.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:07.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:41:07.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:07.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:07.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:07.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:07.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.120+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:07.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:07.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:07.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:41:07.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:41:07.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:07.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:07.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000bc00 con 0x7f6400146050 2021-03-09T09:41:07.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:07.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:07.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:41:07.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:41:07.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.121+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:07.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.122+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012b10 secure :-1 s=READY pgs=8681 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:07.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.122+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:41:07.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.122+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:07.136 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:07.123+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23711 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:07.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:41:07.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.124+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:07.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.124+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:07.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.124+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:07.138 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:07.139 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:07.139 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:07.139 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:07.139 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:07.140 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:07.140 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:07.140 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:07.140 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:07.141 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:07.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f6400146050 2021-03-09T09:41:07.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.124+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:07.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.124+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:41:07.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:41:07.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:07.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:07.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:08.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:08.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:08.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:08.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:08.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:08.491+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:09.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:09.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:09.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:09.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:09.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:09.951+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:10.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:10.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:10.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:10.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:11.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:11.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:11.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:12.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:12.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012b10 secure :-1 s=READY pgs=8681 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012b10 secure :-1 s=READY pgs=8681 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f004dd90).stop 2021-03-09T09:41:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:12.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0012b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:12.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0012b10 unknown :-1 s=CLOSED pgs=8681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:12.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:41:12.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:12.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:12.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:12.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:12.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:12.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:12.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:12.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00091f0 con 0x7f63d8003400 2021-03-09T09:41:12.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:12.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ae30 con 0x7f63d8008840 2021-03-09T09:41:12.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:12.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024800 con 0x7f63d8005e00 2021-03-09T09:41:12.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:12.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:12.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:12.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:12.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:12.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.123+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:12.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:41:12.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:12.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:12.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:12.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:12.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:12.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:12.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:12.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:12.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:12.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:12.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:41:12.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:41:12.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:12.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:12.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00018d0 con 0x7f63d8005e00 2021-03-09T09:41:12.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.124+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:12.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.125+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:12.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.125+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010430 con 0x7f63d8005e00 2021-03-09T09:41:12.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.125+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:41:12.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.125+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:12.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.125+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8696 cs=0 l=1 rev1=1 rx=0x7f63f401fd70 tx=0x7f63f4003bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:12.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.125+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:41:12.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.125+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:12.139 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:12.127+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23749 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:12.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40223d0 con 0x7f63d8005e00 2021-03-09T09:41:12.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.127+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:12.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.127+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:12.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.127+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:12.140 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:12.140 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:12.141 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:12.141 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:12.141 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:12.141 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:12.141 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:12.142 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:12.142 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:12.142 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:12.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022530 con 0x7f63d8005e00 2021-03-09T09:41:12.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.127+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:12.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.127+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:41:12.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63d8005e00 2021-03-09T09:41:12.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:12.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:12.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:13.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:13.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:13.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:13.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:14.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:14.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:14.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:14.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:14.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:14.952+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:15.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:15.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:15.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:15.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:16.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:16.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:17.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:17.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:17.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8696 cs=0 l=1 rev1=1 rx=0x7f63f401fd70 tx=0x7f63f4003bb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:17.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8696 cs=0 l=1 rev1=1 rx=0x7f63f401fd70 tx=0x7f63f4003bb0).stop 2021-03-09T09:41:17.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:17.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:17.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:17.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:17.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:41:17.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:17.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:17.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:17.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:17.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00128a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:17.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:17.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:17.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018220 con 0x7f63d8003400 2021-03-09T09:41:17.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:17.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:41:17.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:17.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015f80 con 0x7f6400146050 2021-03-09T09:41:17.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.127+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:17.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:17.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:17.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:17.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:17.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:17.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:41:17.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:17.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:17.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:17.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:17.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.128+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:17.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:17.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:17.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:41:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:41:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018fd0 con 0x7f6400146050 2021-03-09T09:41:17.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:17.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:17.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000bc00 con 0x7f6400146050 2021-03-09T09:41:17.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:41:17.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.129+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:17.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.130+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:41:17.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.130+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:17.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.130+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00128a0 secure :-1 s=READY pgs=8711 cs=0 l=1 rev1=1 rx=0x7f63f0048070 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:17.143 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:17.130+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23788 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:17.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.132+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007fc80 con 0x7f6400146050 2021-03-09T09:41:17.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.132+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:17.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.132+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:17.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.132+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:17.144 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:17.145 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:17.145 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:17.145 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:17.145 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:17.146 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:17.146 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:17.146 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:17.146 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:17.146 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:17.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:41:17.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.134+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:17.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.134+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:41:17.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:41:17.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:17.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:17.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:18.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:18.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:18.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:18.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:18.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:18.493+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:19.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:19.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:19.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:19.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:19.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:19.953+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:20.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:20.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:20.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:20.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:21.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:21.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:21.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:21.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:22.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:22.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:22.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00128a0 secure :-1 s=READY pgs=8711 cs=0 l=1 rev1=1 rx=0x7f63f0048070 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00128a0 secure :-1 s=READY pgs=8711 cs=0 l=1 rev1=1 rx=0x7f63f0048070 tx=0x7f63f0006610).stop 2021-03-09T09:41:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00128a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:22.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00128a0 unknown :-1 s=CLOSED pgs=8711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:41:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:22.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:22.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:22.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:22.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:22.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d310 con 0x7f63d8003400 2021-03-09T09:41:22.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:22.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067830 con 0x7f63d8008840 2021-03-09T09:41:22.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:22.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8005e00 2021-03-09T09:41:22.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:22.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:22.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:22.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:22.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:22.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:22.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:41:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.131+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.132+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.132+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:22.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.132+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.132+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.132+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.132+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:22.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:41:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:41:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:22.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b600 con 0x7f63d8005e00 2021-03-09T09:41:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016ed0 con 0x7f63d8005e00 2021-03-09T09:41:22.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:41:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:41:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:22.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.133+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8722 cs=0 l=1 rev1=1 rx=0x7f63fc071320 tx=0x7f63fc051f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:22.146 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:22.134+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23826 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:22.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63d8005e00 2021-03-09T09:41:22.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.134+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:22.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.134+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:22.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.134+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:22.148 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:22.148 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:22.148 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:22.148 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:22.148 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:22.149 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:22.149 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:41:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.135+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.135+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:41:22.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.136+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63d8005e00 2021-03-09T09:41:22.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:22.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:22.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:23.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:23.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:23.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:23.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:24.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:24.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:24.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:24.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:24.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:24.954+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:25.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:25.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:25.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:25.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:26.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:26.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:26.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:26.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:27.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:27.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:27.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8722 cs=0 l=1 rev1=1 rx=0x7f63fc071320 tx=0x7f63fc051f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:27.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8722 cs=0 l=1 rev1=1 rx=0x7f63fc071320 tx=0x7f63fc051f60).stop 2021-03-09T09:41:27.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:27.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:27.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:27.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:27.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:41:27.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:27.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:27.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:27.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:27.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:27.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:27.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:27.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T09:41:27.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:27.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024800 con 0x7f63d8008840 2021-03-09T09:41:27.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:27.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00091f0 con 0x7f6400146050 2021-03-09T09:41:27.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:27.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:27.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:27.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:27.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.135+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:27.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:27.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:41:27.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:27.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:27.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:27.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:27.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:27.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:27.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:27.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:27.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:27.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:27.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:41:27.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:41:27.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:27.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:27.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:41:27.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:27.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:27.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018fd0 con 0x7f6400146050 2021-03-09T09:41:27.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:41:27.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.136+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:27.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.137+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:41:27.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.137+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:27.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132c0 secure :-1 s=READY pgs=8738 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f0042fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:27.151 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:27.138+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23864 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:27.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.139+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007fc80 con 0x7f6400146050 2021-03-09T09:41:27.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.139+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:27.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.139+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:27.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.139+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:27.152 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:27.153 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:27.153 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:27.153 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:27.153 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:27.153 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:27.154 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:27.154 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:27.154 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:27.154 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:27.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.139+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:41:27.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.139+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:27.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.139+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:41:27.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.140+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042320 con 0x7f6400146050 2021-03-09T09:41:27.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:27.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:27.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:28.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:28.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:28.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:28.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:28.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:28.494+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:29.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:29.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:29.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:29.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:29.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:29.955+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:30.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:30.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:30.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:30.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:31.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:31.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:31.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:31.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:32.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00132c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:32.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00132c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:32.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132c0 secure :-1 s=READY pgs=8738 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f0042fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:32.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132c0 secure :-1 s=READY pgs=8738 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f0042fa0).stop 2021-03-09T09:41:32.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:32.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:32.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00132c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:32.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00132c0 unknown :-1 s=CLOSED pgs=8738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:32.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:41:32.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:32.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:32.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.138+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:32.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:32.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:32.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:32.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:32.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:41:32.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:32.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f970 con 0x7f63d8008840 2021-03-09T09:41:32.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:32.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00151e0 con 0x7f63d8005e00 2021-03-09T09:41:32.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:32.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:32.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:32.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:32.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:32.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:32.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:41:32.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:32.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:32.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:32.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:32.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.139+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:32.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:32.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:32.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:32.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:32.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:32.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:41:32.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:41:32.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:32.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:32.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T09:41:32.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:32.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:32.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:41:32.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:41:32.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.140+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:32.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.141+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:41:32.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.141+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8750 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f00158b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:32.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.141+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:32.154 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:32.142+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23901 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:32.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00884c0 con 0x7f63d8005e00 2021-03-09T09:41:32.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.142+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:32.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.142+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:32.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.142+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:32.156 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:32.156 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:32.156 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:32.156 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:32.156 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:32.157 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:32.157 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:32.157 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:32.157 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:32.157 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:32.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T09:41:32.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.142+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:32.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.142+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:41:32.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f000fc20 con 0x7f63d8005e00 2021-03-09T09:41:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:32.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:33.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:33.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:33.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:33.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:34.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:34.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:34.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:34.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:34.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:34.956+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:35.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:35.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:35.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:35.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:36.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:36.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:36.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:36.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:37.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:37.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:37.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8750 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f00158b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:37.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8750 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f00158b0).stop 2021-03-09T09:41:37.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:37.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:37.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:37.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8750 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:37.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:41:37.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:37.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:37.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00251a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:37.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:37.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:37.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:37.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:37.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c990 con 0x7f63d8003400 2021-03-09T09:41:37.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:37.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067e20 con 0x7f63d8008840 2021-03-09T09:41:37.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:37.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c9a0 con 0x7f6400146050 2021-03-09T09:41:37.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:37.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.142+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.143+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:37.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:37.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.143+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00251a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.143+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:41:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.143+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.143+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.143+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:37.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.143+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:37.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.143+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:37.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:37.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:37.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:37.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:37.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:37.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:41:37.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:41:37.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:37.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:37.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:41:37.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:37.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:37.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b600 con 0x7f6400146050 2021-03-09T09:41:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:41:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.144+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.145+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:41:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.145+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:37.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.145+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00251a0 secure :-1 s=READY pgs=8764 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:37.158 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:37.145+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23941 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:37.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:41:37.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.147+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:37.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.147+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:37.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.147+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:37.160 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:37.160 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:37.160 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:37.160 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:37.160 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:37.161 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:37.161 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:37.161 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:37.161 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:37.161 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:37.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:41:37.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.149+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:37.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.150+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:41:37.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.151+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc023d30 con 0x7f6400146050 2021-03-09T09:41:37.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:37.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:37.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:38.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:38.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:38.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:38.495+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:39.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:39.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:39.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:39.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:39.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:39.957+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:40.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:40.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:40.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:41.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:41.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:41.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:41.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:42.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00251a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:42.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00251a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:42.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00251a0 secure :-1 s=READY pgs=8764 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:42.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00251a0 secure :-1 s=READY pgs=8764 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009a70).stop 2021-03-09T09:41:42.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:42.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:42.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00251a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:42.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00251a0 unknown :-1 s=CLOSED pgs=8764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:42.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:41:42.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:42.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:42.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:42.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:42.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:42.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:42.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:42.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015f80 con 0x7f63d8003400 2021-03-09T09:41:42.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:42.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:41:42.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:42.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b4b0 con 0x7f63d8005e00 2021-03-09T09:41:42.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:42.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:42.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:42.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:42.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:42.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:42.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:41:42.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:42.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:42.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:42.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:42.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.146+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:42.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:42.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:42.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:42.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:42.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:42.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:41:42.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:41:42.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:42.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:42.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:41:42.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:42.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:42.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T09:41:42.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:41:42.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:42.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:41:42.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.148+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:42.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.147+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8779 cs=0 l=1 rev1=1 rx=0x7f63e0015ef0 tx=0x7f63f007ef50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:42.163 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:42.149+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 23980 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:42.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000eef0 con 0x7f63d8005e00 2021-03-09T09:41:42.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.150+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:42.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.150+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:42.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.150+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:42.165 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:42.165 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:42.165 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:42.165 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:42.166 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:42.166 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:42.166 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:42.166 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:42.167 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:42.167 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:42.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:41:42.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.150+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:42.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.150+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:41:42.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.151+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8005e00 2021-03-09T09:41:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:42.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:43.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:43.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:43.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:43.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:44.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:44.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:44.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:44.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:44.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:44.958+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:45.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:45.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:45.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:45.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:46.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:46.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:46.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:46.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:47.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:47.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:47.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8779 cs=0 l=1 rev1=1 rx=0x7f63e0015ef0 tx=0x7f63f007ef50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:47.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8779 cs=0 l=1 rev1=1 rx=0x7f63e0015ef0 tx=0x7f63f007ef50).stop 2021-03-09T09:41:47.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:47.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:47.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:47.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:47.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:41:47.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:47.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:47.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:47.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:47.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:47.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:47.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:47.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00151e0 con 0x7f63d8003400 2021-03-09T09:41:47.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:47.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:41:47.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:47.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b620 con 0x7f6400146050 2021-03-09T09:41:47.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:47.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:47.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:47.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:47.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.150+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:47.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:47.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:41:47.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:47.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:47.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:47.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:47.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:47.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:47.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:47.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:47.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:47.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:47.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:41:47.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:41:47.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:47.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:47.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:41:47.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:47.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:47.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:41:47.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:41:47.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.151+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:47.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.152+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:41:47.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.152+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:47.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.152+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027cf0 secure :-1 s=READY pgs=8792 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc03de70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:47.165 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:47.154+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24018 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:47.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.154+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:41:47.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.154+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:47.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.154+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:47.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.154+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:47.166 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:47.167 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:47.167 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:47.167 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:47.167 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:47.167 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:47.168 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:47.168 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:47.168 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:47.168 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:47.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.154+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:41:47.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.154+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:47.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.154+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:41:47.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:41:47.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:47.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:47.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:48.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:48.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:48.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:48.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:48.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:48.497+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:49.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:49.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:49.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:49.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:49.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:49.959+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:50.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:50.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:50.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:50.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:51.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:51.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:51.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:51.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:52.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:41:52.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:52.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027cf0 secure :-1 s=READY pgs=8792 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc03de70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:52.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027cf0 secure :-1 s=READY pgs=8792 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc03de70).stop 2021-03-09T09:41:52.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:52.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:52.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:52.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0027cf0 unknown :-1 s=CLOSED pgs=8792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:52.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:41:52.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:52.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:52.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:52.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:52.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:52.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:52.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:52.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:41:52.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:52.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00151e0 con 0x7f63d8008840 2021-03-09T09:41:52.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:52.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fe80 con 0x7f63d8005e00 2021-03-09T09:41:52.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:52.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:52.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.154+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:52.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:52.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:52.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.155+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:52.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.155+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:41:52.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.155+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:52.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.155+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:52.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.156+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:52.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.156+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:52.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.156+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:52.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:52.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:52.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:52.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:52.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:52.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:41:52.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:41:52.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:52.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:52.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b600 con 0x7f63d8005e00 2021-03-09T09:41:52.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:52.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:52.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00018d0 con 0x7f63d8005e00 2021-03-09T09:41:52.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:41:52.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:52.169 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:52.157+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24057 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:52.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.158+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8813 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:52.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.158+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:41:52.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.158+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:52.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.159+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400dd40 con 0x7f63d8005e00 2021-03-09T09:41:52.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.159+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:52.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.159+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:52.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.159+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:52.172 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:52.172 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:52.172 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:52.172 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:52.173 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:52.173 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:52.173 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:52.173 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:52.174 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:52.174 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:52.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.161+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63d8005e00 2021-03-09T09:41:52.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.161+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:52.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.161+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:41:52.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400dd40 con 0x7f63d8005e00 2021-03-09T09:41:52.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:52.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:52.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:53.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:53.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:53.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:53.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:54.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:54.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:54.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:54.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:54.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:54.960+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:41:55.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:55.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:55.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:55.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:56.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:56.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:56.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:56.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:57.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:41:57.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:41:57.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8813 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:41:57.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8813 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4017ed0).stop 2021-03-09T09:41:57.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:57.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:41:57.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:41:57.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:41:57.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:41:57.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:57.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:41:57.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:57.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:41:57.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:41:57.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:41:57.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:57.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T09:41:57.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:57.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:41:57.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:41:57.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fc30 con 0x7f6400146050 2021-03-09T09:41:57.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:41:57.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:41:57.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.158+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:57.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.159+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:41:57.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.159+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:41:57.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.159+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:41:57.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.159+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:41:57.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.159+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:41:57.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.159+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:41:57.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.159+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:41:57.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.159+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:41:57.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.159+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:41:57.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:41:57.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:57.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:57.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:41:57.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:41:57.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:41:57.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:41:57.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:57.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:57.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f6400146050 2021-03-09T09:41:57.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:41:57.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:41:57.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:41:57.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:41:57.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.160+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:57.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.161+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:41:57.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.161+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:41:57.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.161+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cd80 secure :-1 s=READY pgs=8826 cs=0 l=1 rev1=1 rx=0x7f63e0069060 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:41:57.175 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:41:57.161+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24095 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:41:57.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066ef0 con 0x7f6400146050 2021-03-09T09:41:57.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.162+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:41:57.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.162+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:41:57.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.162+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:41:57.177 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:41:57.177 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:41:57.178 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:41:57.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc066ef0 con 0x7f6400146050 2021-03-09T09:41:57.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.164+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:41:57.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.164+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:41:57.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:41:57.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:57.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:57.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:58.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:58.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:58.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:58.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:58.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:58.498+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:41:59.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:59.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:41:59.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:59.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:41:59.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:41:59.961+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:00.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:00.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:00.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:00.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:01.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:01.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:01.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:01.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:02.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:02.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:02.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cd80 secure :-1 s=READY pgs=8826 cs=0 l=1 rev1=1 rx=0x7f63e0069060 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:02.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cd80 secure :-1 s=READY pgs=8826 cs=0 l=1 rev1=1 rx=0x7f63e0069060 tx=0x7f63fc04a740).stop 2021-03-09T09:42:02.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:02.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:02.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:02.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cd80 unknown :-1 s=CLOSED pgs=8826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:02.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:42:02.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:02.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:02.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:02.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:02.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:02.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:02.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:02.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:42:02.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:02.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b310 con 0x7f63d8008840 2021-03-09T09:42:02.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:02.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067830 con 0x7f63d8005e00 2021-03-09T09:42:02.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:02.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:02.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:02.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:02.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.162+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:02.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.163+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:02.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.163+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:42:02.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.163+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:02.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.163+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:02.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.163+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:02.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.163+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:02.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.163+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:02.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:02.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:42:02.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:42:02.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:02.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:02.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8005e00 2021-03-09T09:42:02.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:02.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:02.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f63d8005e00 2021-03-09T09:42:02.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:42:02.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.164+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:02.177 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:02.165+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24134 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.165+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8838 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc0327c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.165+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:42:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.165+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:02.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fd30 con 0x7f63d8005e00 2021-03-09T09:42:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.167+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.167+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.167+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:02.180 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:02.181 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:02.182 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:02.182 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:02.182 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:02.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc032520 con 0x7f63d8005e00 2021-03-09T09:42:02.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.170+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:02.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.170+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:42:02.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.171+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06d170 con 0x7f63d8005e00 2021-03-09T09:42:02.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:02.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:02.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:03.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:03.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:04.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:04.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:04.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:04.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:04.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:04.962+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:05.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:05.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:05.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:05.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:06.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:06.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:06.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:06.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:07.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8838 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc0327c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:07.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8838 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc0327c0).stop 2021-03-09T09:42:07.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:07.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:07.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:07.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:07.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:42:07.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:07.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:07.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:07.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:07.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:07.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:07.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:07.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c990 con 0x7f63d8003400 2021-03-09T09:42:07.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:07.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00172e0 con 0x7f63d8008840 2021-03-09T09:42:07.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:07.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017e30 con 0x7f6400146050 2021-03-09T09:42:07.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:07.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:07.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:07.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:07.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.166+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:07.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.167+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:07.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.167+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:42:07.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.167+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:07.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.167+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:07.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.167+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:07.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.167+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:07.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.167+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:07.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:07.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:07.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:07.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:07.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:07.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:42:07.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:42:07.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:07.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:07.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018fd0 con 0x7f6400146050 2021-03-09T09:42:07.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:07.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:07.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:42:07.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:42:07.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.168+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:07.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.169+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f110 secure :-1 s=READY pgs=8849 cs=0 l=1 rev1=1 rx=0x7f63e0065290 tx=0x7f63fc03c8a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:07.182 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:07.169+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24181 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:07.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.169+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:42:07.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.169+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:07.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc033330 con 0x7f6400146050 2021-03-09T09:42:07.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.170+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:07.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.170+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:07.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.170+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:07.184 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:07.184 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:07.184 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:07.184 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:07.185 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:07.185 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:07.185 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:07.186 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:07.186 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:07.186 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:07.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05d490 con 0x7f6400146050 2021-03-09T09:42:07.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.170+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:07.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.170+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:42:07.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06d170 con 0x7f6400146050 2021-03-09T09:42:07.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:07.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:07.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:08.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:08.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:08.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:08.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:08.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:08.499+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:09.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:09.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:09.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:09.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:09.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:09.963+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:10.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:10.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:10.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:10.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:11.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:11.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:12.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:12.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:12.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f110 secure :-1 s=READY pgs=8849 cs=0 l=1 rev1=1 rx=0x7f63e0065290 tx=0x7f63fc03c8a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:12.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f110 secure :-1 s=READY pgs=8849 cs=0 l=1 rev1=1 rx=0x7f63e0065290 tx=0x7f63fc03c8a0).stop 2021-03-09T09:42:12.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:12.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:12.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001f110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:12.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001f110 unknown :-1 s=CLOSED pgs=8849 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:12.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:42:12.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:12.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:12.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:12.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:12.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:12.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:12.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:12.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:42:12.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:12.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:42:12.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:12.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b4c0 con 0x7f63d8005e00 2021-03-09T09:42:12.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.169+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:12.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.170+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:12.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.170+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:12.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.170+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:12.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.170+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:12.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.170+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:12.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.170+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:42:12.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.170+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:12.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.170+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:12.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.171+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:12.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.171+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:12.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.171+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:12.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:12.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:12.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:12.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:12.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:12.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:42:12.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:42:12.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:12.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:12.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d30 con 0x7f63d8005e00 2021-03-09T09:42:12.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:12.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:12.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.172+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018fd0 con 0x7f63d8005e00 2021-03-09T09:42:12.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:42:12.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:12.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:42:12.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:12.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8870 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc012f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:12.185 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24220 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:12.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02b710 con 0x7f63d8005e00 2021-03-09T09:42:12.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:12.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:12.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.173+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:12.186 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:12.186 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:12.187 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:12.187 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:12.187 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:12.187 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:12.187 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:12.187 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:12.188 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:12.188 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:12.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.174+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00fe60 con 0x7f63d8005e00 2021-03-09T09:42:12.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.174+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:12.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.174+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:42:12.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T09:42:12.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:12.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:12.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:13.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:13.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:13.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:13.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:14.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:14.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:14.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:14.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:14.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:14.964+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:15.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:15.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:15.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:15.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:16.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:16.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:16.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:16.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:17.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:17.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:17.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8870 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc012f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:17.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8870 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc012f30).stop 2021-03-09T09:42:17.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:17.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:17.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:17.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:17.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:42:17.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:17.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:17.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:17.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:17.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:17.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:17.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:17.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c990 con 0x7f63d8003400 2021-03-09T09:42:17.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:17.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T09:42:17.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:17.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00297b0 con 0x7f6400146050 2021-03-09T09:42:17.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:17.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.173+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:17.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.174+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:17.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.174+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:17.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.174+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:42:17.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.174+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:17.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.174+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:17.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.174+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:17.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.174+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:17.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:17.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:17.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:17.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:17.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:17.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:17.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:17.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:17.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:42:17.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:42:17.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:17.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:17.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:42:17.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:17.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:17.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b600 con 0x7f6400146050 2021-03-09T09:42:17.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:42:17.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.175+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:17.189 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:17.176+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24258 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:17.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.177+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 secure :-1 s=READY pgs=8883 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f4022440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:17.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.177+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:42:17.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.177+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:17.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034ca0 con 0x7f6400146050 2021-03-09T09:42:17.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.178+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:17.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.178+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:17.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.178+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:17.191 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:17.191 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:17.192 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:17.192 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:17.192 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:17.192 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:17.192 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:17.193 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:17.193 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:17.193 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:17.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005480 con 0x7f6400146050 2021-03-09T09:42:17.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.183+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:17.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.183+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:42:17.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4034ca0 con 0x7f6400146050 2021-03-09T09:42:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:17.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:17.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:18.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:18.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:18.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:18.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:18.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:18.501+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:19.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:19.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:19.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:19.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:19.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:19.965+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:20.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:20.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:20.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:20.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:21.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:21.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:21.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:22.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:22.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:22.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 secure :-1 s=READY pgs=8883 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f4022440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:22.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 secure :-1 s=READY pgs=8883 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f4022440).stop 2021-03-09T09:42:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 unknown :-1 s=CLOSED pgs=8883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:42:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:22.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:22.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:22.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:22.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a890 con 0x7f63d8003400 2021-03-09T09:42:22.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:22.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017e30 con 0x7f63d8008840 2021-03-09T09:42:22.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:22.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:42:22.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:22.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:22.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.177+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:22.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.178+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:22.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.178+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:22.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.178+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:22.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.178+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:42:22.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.178+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:22.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.178+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:22.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.178+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:22.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.178+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:22.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.178+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:22.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:22.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:22.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:22.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:22.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:22.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:42:22.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:42:22.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:22.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:22.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d800 con 0x7f63d8005e00 2021-03-09T09:42:22.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:22.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:22.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8005e00 2021-03-09T09:42:22.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:42:22.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.179+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:22.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.180+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8890 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f405bcd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:22.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.180+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:42:22.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.180+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:22.193 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:22.180+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24306 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:22.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034de0 con 0x7f63d8005e00 2021-03-09T09:42:22.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.181+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:22.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.181+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:22.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.181+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:22.195 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:22.196 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:22.196 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:22.196 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:22.196 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:22.196 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:22.197 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:22.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005480 con 0x7f63d8005e00 2021-03-09T09:42:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.184+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.184+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:42:22.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404bb50 con 0x7f63d8005e00 2021-03-09T09:42:22.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:22.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:23.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:23.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:23.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:23.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:24.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:24.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:24.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:24.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:24.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:24.966+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:25.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:25.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:25.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:25.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:26.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:26.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:26.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:26.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8890 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f405bcd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8890 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f405bcd0).stop 2021-03-09T09:42:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:27.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:27.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:27.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:42:27.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:27.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:27.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:27.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:27.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:27.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:27.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:27.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b310 con 0x7f63d8003400 2021-03-09T09:42:27.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:27.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce50 con 0x7f63d8008840 2021-03-09T09:42:27.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:27.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00245c0 con 0x7f6400146050 2021-03-09T09:42:27.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:27.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:27.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:27.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.181+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:27.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.182+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:42:27.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.182+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:27.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.182+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.182+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.182+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.182+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.182+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.182+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:27.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:42:27.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:42:27.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:27.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:27.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e900 con 0x7f6400146050 2021-03-09T09:42:27.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:27.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:27.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d800 con 0x7f6400146050 2021-03-09T09:42:27.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:42:27.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:27.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:42:27.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:27.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.183+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 secure :-1 s=READY pgs=8906 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f4072290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:27.197 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:27.184+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24344 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:27.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400d5d0 con 0x7f6400146050 2021-03-09T09:42:27.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.185+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:27.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.185+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:27.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.185+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:27.198 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:27.199 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:27.200 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:27.200 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:27.200 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:27.200 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:27.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400d3a0 con 0x7f6400146050 2021-03-09T09:42:27.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.185+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:27.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.185+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:42:27.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.186+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404bb50 con 0x7f6400146050 2021-03-09T09:42:27.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:27.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:27.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:28.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:28.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:28.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:28.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:28.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:28.502+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:29.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:29.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:29.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:29.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:29.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:29.967+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:30.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:30.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:30.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:30.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:31.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:31.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:31.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:31.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:32.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:32.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:32.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 secure :-1 s=READY pgs=8906 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f4072290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:32.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 secure :-1 s=READY pgs=8906 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f4072290).stop 2021-03-09T09:42:32.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:32.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:32.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 unknown :-1 s=CLOSED pgs=8906 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:42:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0016860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:32.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:32.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:32.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:32.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:32.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T09:42:32.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:32.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00297b0 con 0x7f63d8008840 2021-03-09T09:42:32.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:32.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T09:42:32.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.184+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:32.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.185+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:32.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.185+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:32.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.185+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:32.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.185+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:32.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.185+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0016860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:32.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.185+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:42:32.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.185+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:32.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.185+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:32.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:32.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:32.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:32.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:42:32.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:42:32.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:32.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:32.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001f810 con 0x7f63d8005e00 2021-03-09T09:42:32.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:32.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:32.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e900 con 0x7f63d8005e00 2021-03-09T09:42:32.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:42:32.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.186+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:32.199 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:32.187+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24383 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:32.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.188+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0016860 secure :-1 s=READY pgs=8920 cs=0 l=1 rev1=1 rx=0x7f63f4073070 tx=0x7f63f405b650).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.188+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:42:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.188+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405bea0 con 0x7f63d8005e00 2021-03-09T09:42:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.188+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.188+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:32.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.188+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:32.202 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:32.202 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:32.202 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:32.203 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:32.203 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:32.203 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:32.203 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:32.203 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:32.204 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:32.204 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:32.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405bea0 con 0x7f63d8005e00 2021-03-09T09:42:32.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.190+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:32.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.190+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:42:32.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404b440 con 0x7f63d8005e00 2021-03-09T09:42:32.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:32.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:32.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:33.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:33.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:33.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:33.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:34.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:34.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:34.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:34.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:34.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:34.968+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:35.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:35.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:35.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:35.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:36.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:36.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:36.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:37.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0016860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:37.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0016860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:37.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0016860 secure :-1 s=READY pgs=8920 cs=0 l=1 rev1=1 rx=0x7f63f4073070 tx=0x7f63f405b650).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:37.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0016860 secure :-1 s=READY pgs=8920 cs=0 l=1 rev1=1 rx=0x7f63f4073070 tx=0x7f63f405b650).stop 2021-03-09T09:42:37.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:37.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:37.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0016860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:37.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0016860 unknown :-1 s=CLOSED pgs=8920 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:37.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:42:37.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:37.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:37.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:37.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:37.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:37.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:37.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:37.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:42:37.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:37.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067830 con 0x7f63d8008840 2021-03-09T09:42:37.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:37.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001eb00 con 0x7f6400146050 2021-03-09T09:42:37.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.188+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.189+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.189+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:37.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.189+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:42:37.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.189+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:37.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.189+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:37.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:37.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:37.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:37.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:37.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:37.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:37.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:37.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:37.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:42:37.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:42:37.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:37.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:37.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:42:37.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:37.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:37.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:42:37.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.191+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:42:37.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.191+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:37.203 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:37.191+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24421 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:37.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.192+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 secure :-1 s=READY pgs=8933 cs=0 l=1 rev1=1 rx=0x7f63fc03fcc0 tx=0x7f63fc0327f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:37.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.192+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:42:37.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.192+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:37.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fcf0 con 0x7f6400146050 2021-03-09T09:42:37.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.192+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:37.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.192+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:37.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.192+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:37.206 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:37.206 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:37.206 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:37.208 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:37.208 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:37.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T09:42:37.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.194+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:37.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.194+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:42:37.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:42:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:37.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:37.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:38.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:38.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:38.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:38.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:38.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:38.504+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:39.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:39.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:39.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:39.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:39.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:39.969+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:40.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:40.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:40.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:40.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:41.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:41.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:41.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:41.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:42.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:42:42.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:42.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 secure :-1 s=READY pgs=8933 cs=0 l=1 rev1=1 rx=0x7f63fc03fcc0 tx=0x7f63fc0327f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:42.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 secure :-1 s=READY pgs=8933 cs=0 l=1 rev1=1 rx=0x7f63fc03fcc0 tx=0x7f63fc0327f0).stop 2021-03-09T09:42:42.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:42.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:42.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002b500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:42.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002b500 unknown :-1 s=CLOSED pgs=8933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:42.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:42:42.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:42.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:42.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:42.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:42.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:42.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:42.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:42.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c990 con 0x7f63d8003400 2021-03-09T09:42:42.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:42.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T09:42:42.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:42.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T09:42:42.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:42.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:42.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:42.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:42.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.192+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:42.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:42.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:42:42.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:42.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:42.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:42.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:42.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:42.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:42.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:42.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:42.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:42.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e002b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:42.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:42:42.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:42:42.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:42.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:42.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006a870 con 0x7f63d8005e00 2021-03-09T09:42:42.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:42.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:42.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8005e00 2021-03-09T09:42:42.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:42:42.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.193+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:42.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.194+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8946 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0589d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:42.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.194+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:42:42.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.194+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:42.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc062b50 con 0x7f63d8005e00 2021-03-09T09:42:42.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.195+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.195+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.195+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:42.210 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:42.210 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:42.210 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:42.210 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:42.210 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:42.211 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:42.211 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:42.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63d8005e00 2021-03-09T09:42:42.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.196+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:42.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.196+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:42:42.212 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:42.195+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24460 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:42.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.197+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc062b50 con 0x7f63d8005e00 2021-03-09T09:42:42.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:42.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:42.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:43.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:43.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:43.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:43.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:44.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:44.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:44.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:44.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:44.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:44.970+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:45.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:45.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:45.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:45.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:46.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:46.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:46.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:46.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:47.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:47.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:47.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8946 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0589d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:47.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8946 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc0589d0).stop 2021-03-09T09:42:47.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:47.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:47.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:47.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:47.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:42:47.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:47.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:47.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:47.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:47.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00281e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:47.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:47.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:47.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:42:47.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:47.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00297b0 con 0x7f63d8008840 2021-03-09T09:42:47.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:47.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:42:47.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:47.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:47.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:47.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:47.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:47.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00281e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:47.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:42:47.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:47.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.196+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:47.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.197+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:47.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.197+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:47.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.197+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:47.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:47.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:47.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:47.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:47.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:47.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:42:47.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:42:47.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:47.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:47.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:42:47.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:47.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:47.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:42:47.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:42:47.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.198+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:47.211 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:47.199+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24498 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:47.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.199+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:42:47.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.199+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:47.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.199+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00281e0 secure :-1 s=READY pgs=8966 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:47.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:42:47.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.200+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:47.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.200+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:47.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.200+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:47.216 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:47.216 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:47.217 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:47.217 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:47.218 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:47.218 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:47.218 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:47.219 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:47.219 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:47.220 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:47.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:42:47.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.202+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:47.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.203+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:42:47.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:42:47.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:47.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:47.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:48.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:48.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:48.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:48.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:48.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:48.505+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:49.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:49.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:49.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:49.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:49.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:49.971+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:50.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:50.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:50.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:50.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:51.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:51.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:51.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:51.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:52.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00281e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:52.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00281e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:52.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00281e0 secure :-1 s=READY pgs=8966 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:52.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00281e0 secure :-1 s=READY pgs=8966 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004db20).stop 2021-03-09T09:42:52.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e00281e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e00281e0 unknown :-1 s=CLOSED pgs=8966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:52.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:42:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:52.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:52.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:52.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:52.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067830 con 0x7f63d8003400 2021-03-09T09:42:52.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:52.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c990 con 0x7f63d8008840 2021-03-09T09:42:52.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:52.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012c80 con 0x7f63d8005e00 2021-03-09T09:42:52.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:52.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:52.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:52.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:52.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:42:52.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:52.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.200+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:52.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:52.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:52.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:52.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:42:52.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:42:52.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:52.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:52.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:42:52.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:52.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:52.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001f810 con 0x7f63d8005e00 2021-03-09T09:42:52.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:42:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.201+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.202+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8974 cs=0 l=1 rev1=1 rx=0x7f63f405c4b0 tx=0x7f63f400d790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.202+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:42:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.202+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:52.218 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:52.203+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24537 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.203+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405c2f0 con 0x7f63d8005e00 2021-03-09T09:42:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.203+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.203+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.203+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:52.220 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:52.222 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:52.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405c2f0 con 0x7f63d8005e00 2021-03-09T09:42:52.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.204+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:52.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.204+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:42:52.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.205+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022420 con 0x7f63d8005e00 2021-03-09T09:42:52.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:52.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:52.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:53.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:53.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:53.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:53.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:54.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:54.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:54.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:54.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:54.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:54.972+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:42:55.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:55.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:55.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:55.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:56.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:56.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:56.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:56.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:57.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.203+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:42:57.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.203+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:42:57.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.203+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8974 cs=0 l=1 rev1=1 rx=0x7f63f405c4b0 tx=0x7f63f400d790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:42:57.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.203+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=8974 cs=0 l=1 rev1=1 rx=0x7f63f405c4b0 tx=0x7f63f400d790).stop 2021-03-09T09:42:57.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:57.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:42:57.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:42:57.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=8974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:42:57.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:42:57.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:57.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:42:57.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:57.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:42:57.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:42:57.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:42:57.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:57.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce50 con 0x7f63d8003400 2021-03-09T09:42:57.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:57.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63d8008840 2021-03-09T09:42:57.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:42:57.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f6400146050 2021-03-09T09:42:57.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:42:57.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:42:57.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:57.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:42:57.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.204+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:42:57.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.205+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:42:57.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.205+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:42:57.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.205+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:42:57.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.205+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:42:57.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.205+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:42:57.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.205+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:42:57.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.205+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:42:57.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:42:57.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:57.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:57.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:42:57.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:42:57.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:42:57.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:42:57.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:57.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:57.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:42:57.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:42:57.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:42:57.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006a870 con 0x7f6400146050 2021-03-09T09:42:57.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:42:57.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.206+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:57.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.207+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf20 secure :-1 s=READY pgs=8990 cs=0 l=1 rev1=1 rx=0x7f63fc00b700 tx=0x7f63fc06f570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:42:57.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.207+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:42:57.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.207+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:42:57.220 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:42:57.207+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24575 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:42:57.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:42:57.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.208+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:42:57.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.208+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:42:57.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.208+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:42:57.222 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:42:57.222 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:42:57.222 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:57.223 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:42:57.223 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:42:57.223 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:42:57.223 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:42:57.223 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:42:57.224 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:42:57.224 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:42:57.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05deb0 con 0x7f6400146050 2021-03-09T09:42:57.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.211+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:42:57.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.212+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:42:57.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.212+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f6400146050 2021-03-09T09:42:57.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:57.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:57.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:58.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:58.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:58.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:58.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:58.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:58.506+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:42:59.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:59.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:42:59.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:59.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:42:59.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:42:59.973+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:00.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:00.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:00.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:00.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:01.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:01.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:01.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:01.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:02.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:02.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:02.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf20 secure :-1 s=READY pgs=8990 cs=0 l=1 rev1=1 rx=0x7f63fc00b700 tx=0x7f63fc06f570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:02.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf20 secure :-1 s=READY pgs=8990 cs=0 l=1 rev1=1 rx=0x7f63fc00b700 tx=0x7f63fc06f570).stop 2021-03-09T09:43:02.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:02.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:02.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006cf20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:02.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cf20 unknown :-1 s=CLOSED pgs=8990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:02.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:43:02.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:02.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:02.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:02.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:02.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:02.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:02.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:02.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c390 con 0x7f63d8003400 2021-03-09T09:43:02.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:02.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:43:02.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:02.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016f20 con 0x7f63d8005e00 2021-03-09T09:43:02.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:02.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:02.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:02.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:02.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.208+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:02.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:02.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.209+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:43:02.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.209+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:02.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.209+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:02.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.210+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:02.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.210+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:02.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.210+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:02.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:02.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:02.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:02.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:02.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:02.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:43:02.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:43:02.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:02.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:02.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T09:43:02.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:02.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:02.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8005e00 2021-03-09T09:43:02.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:43:02.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:02.225 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:02.211+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24614 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:02.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.212+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:43:02.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.212+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:02.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.212+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9010 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0013bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:02.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.212+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T09:43:02.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.212+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:02.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.213+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:02.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.213+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:02.228 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:02.228 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:02.228 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:02.228 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:02.229 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:02.229 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:02.229 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:02.229 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:02.229 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:02.230 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:02.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.214+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T09:43:02.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.214+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:02.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.215+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:43:02.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T09:43:02.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:02.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:02.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:03.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:03.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:03.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:03.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:04.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:04.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:04.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:04.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:04.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:04.974+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:05.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:05.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:05.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:05.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:06.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:06.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:06.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:06.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:07.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:07.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:07.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9010 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0013bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:07.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9010 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0013bf0).stop 2021-03-09T09:43:07.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:07.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:07.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:07.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=9010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:07.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:43:07.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:07.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:07.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:07.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:07.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:07.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:07.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:07.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63d8003400 2021-03-09T09:43:07.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:07.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064eb0 con 0x7f63d8008840 2021-03-09T09:43:07.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:07.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a30 con 0x7f6400146050 2021-03-09T09:43:07.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.213+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:07.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.214+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:07.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.214+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:07.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.214+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:07.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.214+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:07.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.214+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:07.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.214+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:43:07.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.214+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:07.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.214+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:07.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:07.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:07.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:07.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:07.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:07.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:07.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:07.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:07.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:43:07.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:43:07.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:07.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:07.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:43:07.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:07.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:07.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:43:07.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:43:07.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.215+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:07.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 secure :-1 s=READY pgs=9020 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:07.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.216+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:43:07.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.216+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:07.229 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:07.216+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24652 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:07.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.217+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02ceb0 con 0x7f6400146050 2021-03-09T09:43:07.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.217+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:07.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.217+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:07.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.217+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:07.230 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:07.230 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:07.231 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:07.231 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:07.231 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:07.231 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:07.232 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:07.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02ceb0 con 0x7f6400146050 2021-03-09T09:43:07.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.222+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:07.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.222+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:43:07.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f6400146050 2021-03-09T09:43:07.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:07.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:07.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:08.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:08.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:08.508+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:09.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:09.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:09.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:09.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:09.975+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:10.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:10.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:10.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:10.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:11.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:11.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:11.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:11.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:12.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 secure :-1 s=READY pgs=9020 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 secure :-1 s=READY pgs=9020 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc003a50).stop 2021-03-09T09:43:12.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:12.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:12.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:12.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 unknown :-1 s=CLOSED pgs=9020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:12.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:43:12.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:12.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:12.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:12.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:12.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T09:43:12.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:43:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T09:43:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.216+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:12.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.217+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.217+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.217+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:12.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.217+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:12.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.217+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.217+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:43:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.217+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:12.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.217+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:12.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:12.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:12.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:12.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:12.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:43:12.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:43:12.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:12.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:12.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005dd10 con 0x7f63d8005e00 2021-03-09T09:43:12.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:12.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:12.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T09:43:12.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:43:12.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.218+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:12.231 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:12.220+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24691 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:12.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.220+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:43:12.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.220+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9035 cs=0 l=1 rev1=1 rx=0x7f63f005fe70 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:12.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.220+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:12.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005fee0 con 0x7f63d8005e00 2021-03-09T09:43:12.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.223+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:12.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.223+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:12.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.223+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:12.233 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:12.234 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:12.234 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:12.234 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:12.234 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:12.235 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:12.235 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:12.235 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:12.235 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:12.235 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:12.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8005e00 2021-03-09T09:43:12.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.223+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:12.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.223+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:43:12.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0015750 con 0x7f63d8005e00 2021-03-09T09:43:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:12.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:12.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:13.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:13.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:13.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:13.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:14.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:14.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:14.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:14.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:14.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:14.976+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:15.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:15.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:15.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:15.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:16.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:16.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:16.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:16.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:17.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:43:17.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:17.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9035 cs=0 l=1 rev1=1 rx=0x7f63f005fe70 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:17.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9035 cs=0 l=1 rev1=1 rx=0x7f63f005fe70 tx=0x7f63f004de00).stop 2021-03-09T09:43:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:17.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:17.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:17.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=9035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:17.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:43:17.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:17.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:17.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:17.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:17.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:17.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:17.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:17.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064eb0 con 0x7f63d8003400 2021-03-09T09:43:17.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:17.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f0b0 con 0x7f63d8008840 2021-03-09T09:43:17.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:17.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f6400146050 2021-03-09T09:43:17.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:17.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:17.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:17.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:17.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.220+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:17.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.221+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:17.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.221+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:43:17.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.221+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:17.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.221+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:17.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.221+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:17.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.221+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:17.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.221+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:17.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:17.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:17.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:17.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:17.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:17.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:43:17.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:43:17.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:17.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:17.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00245c0 con 0x7f6400146050 2021-03-09T09:43:17.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:17.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:17.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005dd10 con 0x7f6400146050 2021-03-09T09:43:17.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:43:17.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.222+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:17.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.223+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b340 secure :-1 s=READY pgs=9047 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f005ea20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:17.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.223+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:43:17.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.223+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:17.236 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:17.223+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24731 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:17.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005fea0 con 0x7f6400146050 2021-03-09T09:43:17.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.224+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:17.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.224+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:17.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.224+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:17.238 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:17.238 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:17.238 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:17.238 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:17.239 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:17.239 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:17.239 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:17.240 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:17.240 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:17.240 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:17.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.225+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005fea0 con 0x7f6400146050 2021-03-09T09:43:17.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.225+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:17.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.225+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:43:17.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028bc0 con 0x7f6400146050 2021-03-09T09:43:17.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:17.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:17.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:18.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:18.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:18.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:18.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:18.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:18.509+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:19.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:19.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:19.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:19.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:19.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:19.977+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:20.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:20.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:20.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:20.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:21.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:21.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:21.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:21.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:22.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:22.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:22.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b340 secure :-1 s=READY pgs=9047 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f005ea20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:22.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b340 secure :-1 s=READY pgs=9047 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f005ea20).stop 2021-03-09T09:43:22.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:22.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:22.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001b340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:22.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001b340 unknown :-1 s=CLOSED pgs=9047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:22.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:43:22.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0066080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:22.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:22.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012e80 con 0x7f63d8003400 2021-03-09T09:43:22.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:22.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a30 con 0x7f63d8008840 2021-03-09T09:43:22.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:22.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8005e00 2021-03-09T09:43:22.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:22.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:22.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:22.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:22.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:22.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0066080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:22.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:43:22.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:22.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.224+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:22.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:22.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:22.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:22.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:22.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:22.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:22.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:22.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:22.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:43:22.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:43:22.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:22.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:22.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T09:43:22.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:22.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:22.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:43:22.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:43:22.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.225+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:22.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.226+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:43:22.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.226+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:22.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.226+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0066080 secure :-1 s=READY pgs=9057 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f401ae70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:22.238 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:22.227+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24770 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:22.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4017ee0 con 0x7f63d8005e00 2021-03-09T09:43:22.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.228+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:22.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.228+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:22.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.228+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:22.239 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:22.239 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:22.240 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:22.240 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:22.240 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:22.240 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:22.240 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:22.241 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:22.241 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:22.241 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:22.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f63d8005e00 2021-03-09T09:43:22.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.232+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:22.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400d1c0 con 0x7f63d8005e00 2021-03-09T09:43:22.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.232+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:43:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:22.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:23.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:23.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:23.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:23.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:24.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:24.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:24.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:24.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:24.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:24.978+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:25.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:25.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:25.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:25.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:26.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:26.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:26.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:26.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:27.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0066080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:27.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0066080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:27.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0066080 secure :-1 s=READY pgs=9057 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f401ae70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:27.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0066080 secure :-1 s=READY pgs=9057 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f401ae70).stop 2021-03-09T09:43:27.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:27.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:27.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0066080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:27.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0066080 unknown :-1 s=CLOSED pgs=9057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:27.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:43:27.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:27.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:27.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:27.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:27.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:27.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:27.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:27.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:43:27.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:27.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002cf0 con 0x7f63d8008840 2021-03-09T09:43:27.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:27.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068200 con 0x7f6400146050 2021-03-09T09:43:27.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:27.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:27.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:27.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:27.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:27.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.228+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:27.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:43:27.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:27.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:27.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:27.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:27.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:27.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:27.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:27.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:27.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:27.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:27.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:43:27.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:43:27.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:27.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:27.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a490 con 0x7f6400146050 2021-03-09T09:43:27.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:27.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:27.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:43:27.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:43:27.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.229+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:27.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.230+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:43:27.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.230+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:27.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.230+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 secure :-1 s=READY pgs=9071 cs=0 l=1 rev1=1 rx=0x7f63f40481c0 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:27.243 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:27.231+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24809 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:27.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022870 con 0x7f6400146050 2021-03-09T09:43:27.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.231+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:27.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.231+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:27.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.231+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:27.244 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:27.244 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:27.245 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:27.246 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f6400146050 2021-03-09T09:43:27.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.232+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:27.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.232+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:43:27.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022870 con 0x7f6400146050 2021-03-09T09:43:27.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:27.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:27.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:28.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:28.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:28.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:28.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:28.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:28.510+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:29.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:29.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:29.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:29.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:29.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:29.979+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:30.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:30.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:30.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:30.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:31.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:31.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:31.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:31.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:32.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.230+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:32.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.230+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:32.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.230+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 secure :-1 s=READY pgs=9071 cs=0 l=1 rev1=1 rx=0x7f63f40481c0 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:32.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.230+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 secure :-1 s=READY pgs=9071 cs=0 l=1 rev1=1 rx=0x7f63f40481c0 tx=0x7f63f4017ed0).stop 2021-03-09T09:43:32.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.230+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:32.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.230+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:32.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.230+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000f900 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:32.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.230+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000f900 unknown :-1 s=CLOSED pgs=9071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:32.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.230+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:43:32.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:32.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:32.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:32.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:32.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:32.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:32.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:32.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:43:32.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:32.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064eb0 con 0x7f63d8008840 2021-03-09T09:43:32.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:32.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T09:43:32.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:32.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:32.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:32.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:32.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:32.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.231+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:32.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.232+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:43:32.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.232+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:32.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.232+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:32.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.232+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:32.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.232+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:32.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.232+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:32.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:32.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:32.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:32.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:32.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:32.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:43:32.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:43:32.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:32.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:32.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bee0 con 0x7f63d8005e00 2021-03-09T09:43:32.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:32.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:32.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8005e00 2021-03-09T09:43:32.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:43:32.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:32.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9089 cs=0 l=1 rev1=1 rx=0x7f63e0024b70 tx=0x7f63fc06f570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:32.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:43:32.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.233+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:32.260 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:32.234+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24848 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:32.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02cef0 con 0x7f63d8005e00 2021-03-09T09:43:32.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.234+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:32.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.234+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:32.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.234+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:32.263 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:32.264 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:32.264 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:32.265 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:32.265 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:32.266 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:32.266 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:32.267 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:32.267 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:32.267 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:32.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.236+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02cef0 con 0x7f63d8005e00 2021-03-09T09:43:32.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.236+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:32.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.236+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:43:32.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.238+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc050d80 con 0x7f63d8005e00 2021-03-09T09:43:32.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:32.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:32.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:33.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:33.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:33.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:33.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:34.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:34.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:34.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:34.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:34.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:34.979+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:35.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:35.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:35.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:35.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:36.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:36.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:36.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:36.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:37.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:37.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:37.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9089 cs=0 l=1 rev1=1 rx=0x7f63e0024b70 tx=0x7f63fc06f570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:37.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9089 cs=0 l=1 rev1=1 rx=0x7f63e0024b70 tx=0x7f63fc06f570).stop 2021-03-09T09:43:37.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:37.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:37.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:37.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=9089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:37.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:43:37.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.234+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:37.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:37.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:37.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:37.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:37.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:37.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:37.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8003400 2021-03-09T09:43:37.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:37.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T09:43:37.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:37.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:43:37.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:37.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:37.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.235+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:37.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:37.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.236+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:43:37.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.236+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:37.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.236+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:37.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.236+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:37.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.236+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:37.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.236+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:37.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:37.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:37.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:37.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:37.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:43:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:43:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:37.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f4a0 con 0x7f6400146050 2021-03-09T09:43:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bee0 con 0x7f6400146050 2021-03-09T09:43:37.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:43:37.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.237+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:37.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.238+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 secure :-1 s=READY pgs=9103 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc023b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:37.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.238+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:43:37.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.238+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:37.253 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:37.238+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24886 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:37.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc021460 con 0x7f6400146050 2021-03-09T09:43:37.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.239+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:37.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.239+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:37.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.239+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:37.255 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:37.255 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:37.255 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:37.256 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:37.256 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:37.256 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:37.257 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:37.257 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:37.257 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:37.257 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:37.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.242+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f6400146050 2021-03-09T09:43:37.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.242+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:37.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.242+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:43:37.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021460 con 0x7f6400146050 2021-03-09T09:43:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:37.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:38.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:38.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:38.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:38.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:38.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:38.511+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:39.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:39.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:39.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:39.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:39.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:39.980+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:40.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:40.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:40.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:40.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:41.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:41.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:41.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:41.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:42.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:43:42.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:42.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 secure :-1 s=READY pgs=9103 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc023b50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:42.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 secure :-1 s=READY pgs=9103 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc023b50).stop 2021-03-09T09:43:42.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:42.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:42.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c0e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:42.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 unknown :-1 s=CLOSED pgs=9103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:42.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:43:42.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:42.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:42.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:42.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:42.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:42.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:42.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:42.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008f30 con 0x7f63d8003400 2021-03-09T09:43:42.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:42.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a750 con 0x7f63d8008840 2021-03-09T09:43:42.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:42.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63d8005e00 2021-03-09T09:43:42.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:42.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:42.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:42.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:42.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:42.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:42.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:43:42.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:42.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.239+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:42.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:42.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:42.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:42.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:42.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:42.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:42.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:42.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:42.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:43:42.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:43:42.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:42.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:42.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:43:42.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:42.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:42.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a490 con 0x7f63d8005e00 2021-03-09T09:43:42.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:43:42.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.240+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:42.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.241+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9113 cs=0 l=1 rev1=1 rx=0x7f63f40483f0 tx=0x7f63f4003ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:42.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.241+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:43:42.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.241+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:42.256 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:42.242+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24923 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:42.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401aee0 con 0x7f63d8005e00 2021-03-09T09:43:42.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.243+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:42.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.243+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:42.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.243+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:42.258 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:42.258 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:42.258 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:42.258 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:42.259 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:42.259 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:42.259 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:42.259 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:42.260 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:42.260 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:42.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404deb0 con 0x7f63d8005e00 2021-03-09T09:43:42.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.243+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:42.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.243+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:43:42.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048160 con 0x7f63d8005e00 2021-03-09T09:43:42.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:42.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:42.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:43.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:43.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:43.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:43.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:44.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:44.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:44.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:44.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:44.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:44.981+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:45.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:45.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:45.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:45.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:46.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:46.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:46.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:46.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:47.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.242+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:47.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.242+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:47.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.242+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9113 cs=0 l=1 rev1=1 rx=0x7f63f40483f0 tx=0x7f63f4003ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:47.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.242+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9113 cs=0 l=1 rev1=1 rx=0x7f63f40483f0 tx=0x7f63f4003ad0).stop 2021-03-09T09:43:47.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:47.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:47.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:47.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=9113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:47.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:43:47.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:47.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:47.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:47.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:47.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:47.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:47.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:47.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a450 con 0x7f63d8003400 2021-03-09T09:43:47.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:47.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8008840 2021-03-09T09:43:47.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:47.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f6400146050 2021-03-09T09:43:47.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:47.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:47.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.243+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:47.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.244+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:47.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.244+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:47.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.244+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:47.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.244+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:43:47.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.244+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:47.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.244+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:47.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.245+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:47.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.245+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:47.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.245+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:47.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.245+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:47.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.245+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:47.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.245+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:47.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.245+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:47.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.245+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:47.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:43:47.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:43:47.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:47.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:47.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bfa0 con 0x7f6400146050 2021-03-09T09:43:47.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:47.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:47.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f4a0 con 0x7f6400146050 2021-03-09T09:43:47.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:43:47.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:47.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:43:47.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:47.258 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 24963 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:47.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 secure :-1 s=READY pgs=9132 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc013f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:47.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:43:47.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.247+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:47.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.247+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:47.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.247+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:47.260 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:47.260 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:47.261 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:47.261 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:47.261 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:47.261 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:47.261 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:47.262 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:47.262 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:47.262 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:47.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02c220 con 0x7f6400146050 2021-03-09T09:43:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.249+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.249+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:43:47.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f6400146050 2021-03-09T09:43:47.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:47.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:47.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:48.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:48.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:48.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:48.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:48.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:48.512+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:49.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:49.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:49.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:49.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:49.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:49.982+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:50.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:50.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:50.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:50.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:51.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:51.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:51.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:51.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:52.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 secure :-1 s=READY pgs=9132 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc013f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 secure :-1 s=READY pgs=9132 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc013f20).stop 2021-03-09T09:43:52.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c0e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 unknown :-1 s=CLOSED pgs=9132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:43:52.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:52.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006af00 con 0x7f63d8003400 2021-03-09T09:43:52.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T09:43:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ed00 con 0x7f63d8005e00 2021-03-09T09:43:52.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:52.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:52.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:52.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:52.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:43:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:52.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.247+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.248+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.248+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.248+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:52.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:52.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:43:52.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:43:52.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:52.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:52.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d430 con 0x7f63d8005e00 2021-03-09T09:43:52.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:52.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:52.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00245c0 con 0x7f63d8005e00 2021-03-09T09:43:52.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:43:52.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.249+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:52.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.250+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:43:52.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.250+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9143 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f0009510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:52.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.250+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:52.263 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:52.250+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25000 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:52.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8005e00 2021-03-09T09:43:52.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.251+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:52.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.251+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:52.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.251+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:52.265 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:52.265 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:52.265 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:52.266 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:52.266 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:52.266 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:52.266 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:52.267 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:52.267 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:52.267 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:52.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.252+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T09:43:52.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.252+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:52.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.252+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:43:52.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.253+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T09:43:52.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:52.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:52.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:53.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:53.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:53.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:53.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:54.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:54.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:54.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:54.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:54.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:54.983+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:43:55.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:55.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:55.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:55.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:56.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:56.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:56.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:56.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:57.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:43:57.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:43:57.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9143 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f0009510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:43:57.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9143 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f0009510).stop 2021-03-09T09:43:57.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:57.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:43:57.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:43:57.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=9143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:43:57.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:43:57.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:57.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:43:57.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:57.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:43:57.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:43:57.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:43:57.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:57.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:43:57.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:57.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8008840 2021-03-09T09:43:57.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:43:57.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00153c0 con 0x7f6400146050 2021-03-09T09:43:57.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:43:57.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:43:57.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:57.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:43:57.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:43:57.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:43:57.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:43:57.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:43:57.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:43:57.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:43:57.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:43:57.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.251+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:43:57.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:43:57.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:57.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:57.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:43:57.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:43:57.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:43:57.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:43:57.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:57.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:57.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:43:57.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:43:57.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:43:57.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d430 con 0x7f6400146050 2021-03-09T09:43:57.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:43:57.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:57.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 secure :-1 s=READY pgs=9157 cs=0 l=1 rev1=1 rx=0x7f63e006cdd0 tx=0x7f63f0015890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:43:57.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:43:57.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.253+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:43:57.267 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:43:57.254+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25038 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:43:57.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.255+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f6400146050 2021-03-09T09:43:57.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.255+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:43:57.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.255+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:43:57.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.255+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:43:57.268 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:43:57.269 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:43:57.269 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:57.269 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:43:57.269 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:43:57.269 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:43:57.270 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:43:57.270 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:43:57.270 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:43:57.270 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:43:57.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.258+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007bc80 con 0x7f6400146050 2021-03-09T09:43:57.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.258+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:43:57.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.258+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:43:57.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.260+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00284b0 con 0x7f6400146050 2021-03-09T09:43:57.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:57.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:57.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:58.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:58.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:58.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:58.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:58.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:58.513+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:43:59.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:59.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:43:59.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:59.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:43:59.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:43:59.984+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:00.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:00.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:00.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:00.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:01.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:01.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:01.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:01.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:02.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:02.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:02.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 secure :-1 s=READY pgs=9157 cs=0 l=1 rev1=1 rx=0x7f63e006cdd0 tx=0x7f63f0015890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:02.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 secure :-1 s=READY pgs=9157 cs=0 l=1 rev1=1 rx=0x7f63e006cdd0 tx=0x7f63f0015890).stop 2021-03-09T09:44:02.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:02.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:02.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002c0e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:02.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002c0e0 unknown :-1 s=CLOSED pgs=9157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:02.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:44:02.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:02.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:02.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:02.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:02.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:02.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:02.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:02.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e4b0 con 0x7f63d8003400 2021-03-09T09:44:02.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:02.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8008840 2021-03-09T09:44:02.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:02.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ed00 con 0x7f63d8005e00 2021-03-09T09:44:02.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:02.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:02.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:02.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.254+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:02.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.255+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:02.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.256+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:02.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.256+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:44:02.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.256+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:02.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.256+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:02.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.256+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:02.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.256+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:02.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.256+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:02.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.257+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:02.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.257+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:02.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.257+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:02.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:02.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:02.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:44:02.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:44:02.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:02.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:02.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T09:44:02.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:02.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:02.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:44:02.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:44:02.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:02.269 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25079 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:02.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9177 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404bea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:02.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:44:02.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.258+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:02.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022510 con 0x7f63d8005e00 2021-03-09T09:44:02.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.261+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:02.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.261+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:02.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.261+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:02.272 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:02.272 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:02.272 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:02.273 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:02.273 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:02.273 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:02.273 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:02.274 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:02.274 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:02.274 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:02.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022510 con 0x7f63d8005e00 2021-03-09T09:44:02.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.263+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:02.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.263+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:44:02.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.264+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4018400 con 0x7f63d8005e00 2021-03-09T09:44:02.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:02.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:02.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:03.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:03.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:03.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:03.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:04.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:04.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:04.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:04.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:04.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:04.985+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:05.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:05.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:05.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:05.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:06.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:06.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:06.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:07.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:07.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:07.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9177 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404bea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:07.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9177 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404bea0).stop 2021-03-09T09:44:07.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:07.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:07.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:07.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=9177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:07.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:44:07.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:07.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:07.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:07.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:07.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:07.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:07.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:07.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005eb50 con 0x7f63d8003400 2021-03-09T09:44:07.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:07.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8008840 2021-03-09T09:44:07.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:07.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400146050 2021-03-09T09:44:07.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:07.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:07.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:07.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:07.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.259+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:07.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.260+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:07.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.260+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:44:07.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.260+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:07.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.260+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:07.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.260+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:07.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.260+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:07.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.260+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:07.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:07.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:07.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:07.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:07.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:07.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:44:07.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:44:07.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:07.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:07.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bee0 con 0x7f6400146050 2021-03-09T09:44:07.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:07.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:07.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bfa0 con 0x7f6400146050 2021-03-09T09:44:07.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:44:07.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.261+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:07.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.262+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:44:07.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.262+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9185 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:07.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.262+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:07.274 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:07.262+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25117 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:07.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:44:07.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.263+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:07.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.263+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:07.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.263+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:07.276 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:07.276 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:07.276 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:07.277 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:07.277 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:07.277 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:07.277 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:07.277 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:07.278 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:07.278 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:07.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.266+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:44:07.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.266+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:07.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.266+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:44:07.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T09:44:07.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:07.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:07.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:08.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:08.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:08.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:08.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:08.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:08.514+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:09.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:09.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:09.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:09.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:09.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:09.986+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:10.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:10.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:10.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:10.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:11.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:11.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:11.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:11.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.262+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.262+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.262+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9185 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.262+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9185 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc04a6d0).stop 2021-03-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.262+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.262+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.262+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.262+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=CLOSED pgs=9185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006af00 con 0x7f63d8003400 2021-03-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00692a0 con 0x7f63d8008840 2021-03-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.263+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.264+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.264+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.264+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8005e00 2021-03-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bee0 con 0x7f63d8005e00 2021-03-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9198 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc066780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.266+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.266+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:12.279 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:12.266+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25156 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.266+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.266+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.266+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.266+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.269+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.269+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc00f140 con 0x7f63d8005e00 2021-03-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:12.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:13.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:13.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:13.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:13.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:14.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:14.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:14.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:14.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:14.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:14.987+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:15.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:15.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:15.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:15.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:16.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:16.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:16.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:16.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:17.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.266+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:17.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.266+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:17.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9198 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc066780).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9198 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc066780).stop 2021-03-09T09:44:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:17.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:17.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=9198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:17.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:44:17.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:17.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:17.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:17.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:17.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:17.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:17.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:17.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:44:17.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8008840 2021-03-09T09:44:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022980 con 0x7f6400146050 2021-03-09T09:44:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:17.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:17.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:17.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:44:17.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:17.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:17.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:17.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:17.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:17.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:17.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.267+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:17.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:17.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:17.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:17.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:17.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:17.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:44:17.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:44:17.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:17.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:17.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:44:17.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:17.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:17.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T09:44:17.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:44:17.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:17.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9208 cs=0 l=1 rev1=1 rx=0x7f63f4003ca0 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:17.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.268+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:44:17.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.269+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:17.283 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:17.270+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25194 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:17.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022440 con 0x7f6400146050 2021-03-09T09:44:17.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.270+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:17.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.270+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:17.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.270+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:17.285 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:17.285 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:17.285 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:17.286 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:17.286 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:17.286 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:17.286 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:17.286 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:17.287 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:17.287 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:17.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4018400 con 0x7f6400146050 2021-03-09T09:44:17.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.270+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:17.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.270+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:44:17.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.272+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4034bf0 con 0x7f6400146050 2021-03-09T09:44:17.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:17.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:17.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:18.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:18.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:18.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:18.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:18.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:18.516+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:19.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:19.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:19.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:19.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:19.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:19.988+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:20.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:20.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:20.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:20.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:21.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:21.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:21.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:21.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:22.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:22.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:22.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9208 cs=0 l=1 rev1=1 rx=0x7f63f4003ca0 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:22.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9208 cs=0 l=1 rev1=1 rx=0x7f63f4003ca0 tx=0x7f63f4017ed0).stop 2021-03-09T09:44:22.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:22.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:22.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:22.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=CLOSED pgs=9208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:22.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:44:22.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:22.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:22.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068be0 con 0x7f63d8003400 2021-03-09T09:44:22.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:22.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00692a0 con 0x7f63d8008840 2021-03-09T09:44:22.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:22.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016450 con 0x7f63d8005e00 2021-03-09T09:44:22.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:22.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:22.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:22.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:22.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:22.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:22.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:44:22.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:22.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:22.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:22.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:22.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:22.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:22.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:22.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:22.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:22.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:22.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:44:22.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:44:22.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:22.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:22.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000ce50 con 0x7f63d8005e00 2021-03-09T09:44:22.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:22.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:22.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:44:22.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:44:22.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:22.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:44:22.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:22.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.272+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9221 cs=0 l=1 rev1=1 rx=0x7f63e0025920 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:22.288 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:22.274+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25232 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:22.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:44:22.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.275+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:22.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.275+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:22.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.275+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:22.289 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:22.290 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:22.290 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:22.290 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:22.290 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:22.291 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:22.291 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:22.291 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:22.291 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:22.291 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:22.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.278+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:44:22.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.278+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:22.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.278+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:44:22.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8005e00 2021-03-09T09:44:22.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:22.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:22.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:23.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:23.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:23.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:24.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:24.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:24.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:24.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:24.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:24.989+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:25.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:25.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:25.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:25.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:26.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:26.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:26.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:26.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:27.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.274+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:27.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.274+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:27.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.274+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9221 cs=0 l=1 rev1=1 rx=0x7f63e0025920 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:27.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.274+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9221 cs=0 l=1 rev1=1 rx=0x7f63e0025920 tx=0x7f63f004de00).stop 2021-03-09T09:44:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=9221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:44:27.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:27.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:27.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:27.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:27.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:27.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:27.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:27.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025550 con 0x7f63d8003400 2021-03-09T09:44:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8008840 2021-03-09T09:44:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f6400146050 2021-03-09T09:44:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:27.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:27.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:27.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:27.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:27.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.275+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:27.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:44:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:27.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:27.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:27.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:27.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:27.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:27.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:44:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:44:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d780 con 0x7f6400146050 2021-03-09T09:44:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:44:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:44:27.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.276+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:27.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.277+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:44:27.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.277+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 secure :-1 s=READY pgs=9236 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:27.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.277+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:27.292 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:27.278+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25271 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:27.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c400 con 0x7f6400146050 2021-03-09T09:44:27.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.279+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:27.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.279+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:27.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.279+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:27.294 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:27.294 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:27.294 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:27.295 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:27.295 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:27.295 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:27.295 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:27.295 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:27.296 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:27.296 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:27.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02c220 con 0x7f6400146050 2021-03-09T09:44:27.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.279+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:27.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.279+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:44:27.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.280+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T09:44:27.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:27.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:27.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:28.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:28.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:28.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:28.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:28.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:28.517+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:29.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:29.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:29.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:29.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:29.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:29.990+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:30.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:30.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:31.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:31.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:31.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:31.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:32.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:44:32.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:32.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 secure :-1 s=READY pgs=9236 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:32.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 secure :-1 s=READY pgs=9236 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc003a50).stop 2021-03-09T09:44:32.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:32.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:32.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0019e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:32.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0019e70 unknown :-1 s=CLOSED pgs=9236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:32.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:44:32.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:32.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:32.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:32.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:32.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:32.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:32.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:32.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T09:44:32.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:32.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:44:32.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:32.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8005e00 2021-03-09T09:44:32.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:32.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:32.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:32.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.279+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:32.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.280+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:44:32.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.280+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:32.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.280+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:32.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.280+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:32.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.280+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:32.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:32.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:32.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:32.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:32.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:32.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:32.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:32.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:32.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:44:32.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:44:32.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:32.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:32.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:44:32.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:32.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:32.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:44:32.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:44:32.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.281+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:32.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.282+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9255 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f4003ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:32.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.282+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:44:32.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.282+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:32.301 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:32.283+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25308 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:32.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072eb0 con 0x7f63d8005e00 2021-03-09T09:44:32.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.284+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:32.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.284+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:32.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.284+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:32.303 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:32.303 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:32.303 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:32.304 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:32.304 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:32.304 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:32.305 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:32.305 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:32.305 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:32.306 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:32.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072eb0 con 0x7f63d8005e00 2021-03-09T09:44:32.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.290+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:32.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.290+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:44:32.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.291+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40223e0 con 0x7f63d8005e00 2021-03-09T09:44:32.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:32.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:32.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:33.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:33.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:33.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:33.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:34.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:34.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:34.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:34.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:34.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:34.991+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:35.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:35.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:35.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:35.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:36.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:36.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:36.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:36.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:37.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:37.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:37.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9255 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f4003ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:37.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 secure :-1 s=READY pgs=9255 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f4003ad0).stop 2021-03-09T09:44:37.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:37.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:37.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:37.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c290 unknown :-1 s=CLOSED pgs=9255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:37.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:44:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:37.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:37.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:37.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:37.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:44:37.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:37.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bbb0 con 0x7f63d8008840 2021-03-09T09:44:37.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:37.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e010 con 0x7f6400146050 2021-03-09T09:44:37.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:37.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:37.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:37.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:37.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:37.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:37.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:44:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.284+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:37.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:37.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:37.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:37.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:37.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:37.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:44:37.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:44:37.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:37.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e760 con 0x7f6400146050 2021-03-09T09:44:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000ce50 con 0x7f6400146050 2021-03-09T09:44:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:44:37.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:37.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.285+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9263 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:37.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.286+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:44:37.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.286+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:37.299 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:37.287+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25348 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:37.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:44:37.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.288+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:37.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.288+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:37.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.288+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:37.300 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:37.302 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:37.302 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:37.302 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:37.302 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:37.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f6400146050 2021-03-09T09:44:37.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.288+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:37.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.288+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:44:37.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.289+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e290 con 0x7f6400146050 2021-03-09T09:44:37.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:37.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:37.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:38.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:38.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:38.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:38.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:38.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:38.518+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:39.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:39.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:39.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:39.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:39.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:39.992+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:40.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:40.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:40.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:40.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:41.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:41.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:41.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:41.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:42.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:42.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:42.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9263 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:42.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9263 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006610).stop 2021-03-09T09:44:42.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:42.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:42.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:42.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=CLOSED pgs=9263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:44:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:42.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8003400 2021-03-09T09:44:42.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:42.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:44:42.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:42.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b100 con 0x7f63d8005e00 2021-03-09T09:44:42.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:44:42.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:42.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:42.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:42.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:42.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.288+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:42.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:44:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:44:42.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:42.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:42.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027ad0 con 0x7f63d8005e00 2021-03-09T09:44:42.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:42.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:42.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e760 con 0x7f63d8005e00 2021-03-09T09:44:42.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:44:42.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:42.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c160 secure :-1 s=READY pgs=9277 cs=0 l=1 rev1=1 rx=0x7f63e0068310 tx=0x7f63f0036e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:42.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:44:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.289+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:42.303 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:42.291+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25385 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:44:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.292+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.292+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.292+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:42.306 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:42.306 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:42.306 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:42.306 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:42.307 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:42.307 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:42.307 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:42.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T09:44:42.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.292+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.292+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:44:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00284b0 con 0x7f63d8005e00 2021-03-09T09:44:42.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:42.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:42.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:43.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:43.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:43.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:43.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:44.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:44.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:44.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:44.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:44.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:44.993+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:45.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:45.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:45.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:45.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:46.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:46.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:46.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:46.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:47.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:47.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:47.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c160 secure :-1 s=READY pgs=9277 cs=0 l=1 rev1=1 rx=0x7f63e0068310 tx=0x7f63f0036e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:47.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c160 secure :-1 s=READY pgs=9277 cs=0 l=1 rev1=1 rx=0x7f63e0068310 tx=0x7f63f0036e70).stop 2021-03-09T09:44:47.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c160 unknown :-1 s=CLOSED pgs=9277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:47.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:44:47.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:47.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:47.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:47.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:47.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:47.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:47.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:47.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T09:44:47.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:47.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:44:47.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:47.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018d50 con 0x7f6400146050 2021-03-09T09:44:47.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:47.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:47.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.292+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:47.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.293+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:47.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.293+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:44:47.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.293+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:47.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.293+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:47.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.293+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:47.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.293+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:47.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.293+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:47.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.293+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:47.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.293+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:47.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:47.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:47.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:47.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:47.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:47.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:44:47.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:44:47.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:47.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:47.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e150 con 0x7f6400146050 2021-03-09T09:44:47.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:47.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:47.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d780 con 0x7f6400146050 2021-03-09T09:44:47.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:44:47.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:47.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:44:47.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:47.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.294+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9291 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc075f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:47.307 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:47.296+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25425 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:47.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.296+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:44:47.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.296+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:47.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.296+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:47.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.296+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:47.308 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:47.309 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:47.309 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:47.309 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:47.309 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:47.310 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:47.310 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:47.310 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:47.310 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:47.310 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:47.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.296+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f6400146050 2021-03-09T09:44:47.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.296+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:47.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.296+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:44:47.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.298+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f6400146050 2021-03-09T09:44:47.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:47.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:47.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:48.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:48.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:48.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:48.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:48.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:48.520+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:49.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:49.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:49.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:49.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:49.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:49.994+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:50.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:50.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:50.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:50.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:51.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:51.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:51.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:51.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:52.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:44:52.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:52.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9291 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc075f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:52.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9291 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc075f70).stop 2021-03-09T09:44:52.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:52.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:52.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:52.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=CLOSED pgs=9291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:52.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:44:52.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:52.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:52.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:52.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:52.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00177f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:52.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:52.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:52.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028730 con 0x7f63d8003400 2021-03-09T09:44:52.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:52.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8008840 2021-03-09T09:44:52.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:52.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:44:52.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:52.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:52.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:52.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:52.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.297+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:52.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.298+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00177f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:52.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.298+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:44:52.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.298+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:52.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.298+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:52.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.298+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:52.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.298+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:52.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.298+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:52.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:52.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:52.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:52.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:52.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:52.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:44:52.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:44:52.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:52.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:52.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bee0 con 0x7f63d8005e00 2021-03-09T09:44:52.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:52.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:52.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027ad0 con 0x7f63d8005e00 2021-03-09T09:44:52.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:44:52.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.299+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:52.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.300+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:44:52.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.300+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:52.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.300+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00177f0 secure :-1 s=READY pgs=9308 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f0066f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:52.313 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:52.300+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25462 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.302+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f001ce90 con 0x7f63d8005e00 2021-03-09T09:44:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.302+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:52.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.302+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:52.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.302+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:52.315 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:52.315 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:52.315 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:52.315 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:52.316 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:52.316 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:52.316 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:52.316 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:52.316 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:52.317 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:52.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001ce90 con 0x7f63d8005e00 2021-03-09T09:44:52.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.304+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:52.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.304+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:44:52.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T09:44:52.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:52.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:52.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:53.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:53.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:53.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:53.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:54.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:54.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:54.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:54.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:54.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:54.995+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:44:55.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:55.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:55.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:55.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:56.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:56.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:56.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:56.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:57.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00177f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:44:57.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00177f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:44:57.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00177f0 secure :-1 s=READY pgs=9308 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f0066f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:44:57.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00177f0 secure :-1 s=READY pgs=9308 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f0066f30).stop 2021-03-09T09:44:57.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:57.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:44:57.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00177f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:44:57.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00177f0 unknown :-1 s=CLOSED pgs=9308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:44:57.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:44:57.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:57.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:44:57.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:57.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:44:57.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:44:57.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:44:57.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:57.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63d8003400 2021-03-09T09:44:57.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:57.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8008840 2021-03-09T09:44:57.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:44:57.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ee60 con 0x7f6400146050 2021-03-09T09:44:57.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:44:57.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:44:57.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:57.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:44:57.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:44:57.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:44:57.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:44:57.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:44:57.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:44:57.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:44:57.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:44:57.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:44:57.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:44:57.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:57.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:57.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:44:57.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:44:57.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:44:57.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:44:57.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:57.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:57.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:44:57.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:44:57.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:44:57.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bee0 con 0x7f6400146050 2021-03-09T09:44:57.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:44:57.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:57.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9327 cs=0 l=1 rev1=1 rx=0x7f63f0074040 tx=0x7f63f0003d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:44:57.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:44:57.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.304+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:44:57.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f6400146050 2021-03-09T09:44:57.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.305+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:44:57.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.305+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:44:57.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.305+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:44:57.320 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:44:57.320 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:44:57.321 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:57.321 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:44:57.321 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:44:57.321 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:44:57.321 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:44:57.322 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:44:57.322 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:44:57.322 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:44:57.322 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:44:57.305+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25502 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:44:57.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f006deb0 con 0x7f6400146050 2021-03-09T09:44:57.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.307+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:44:57.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.307+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:44:57.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002d940 con 0x7f6400146050 2021-03-09T09:44:57.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:57.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:57.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:58.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:58.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:58.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:58.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:58.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:58.521+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:44:59.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:59.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:44:59.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:59.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:44:59.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:44:59.996+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:00.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:00.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:00.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:00.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:01.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:01.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:01.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:01.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:02.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:45:02.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:02.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9327 cs=0 l=1 rev1=1 rx=0x7f63f0074040 tx=0x7f63f0003d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:02.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 secure :-1 s=READY pgs=9327 cs=0 l=1 rev1=1 rx=0x7f63f0074040 tx=0x7f63f0003d30).stop 2021-03-09T09:45:02.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:02.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:02.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002bfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:02.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002bfb0 unknown :-1 s=CLOSED pgs=9327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:02.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:45:02.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:02.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.305+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:02.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:02.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:02.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:02.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:02.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:02.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:45:02.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:02.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ae80 con 0x7f63d8008840 2021-03-09T09:45:02.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:02.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065c10 con 0x7f63d8005e00 2021-03-09T09:45:02.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:02.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:02.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:02.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:02.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:02.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:02.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:45:02.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:02.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:02.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:02.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:02.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.306+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:02.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:02.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:02.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:02.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:02.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:02.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:45:02.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:45:02.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:02.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:02.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d4a0 con 0x7f63d8005e00 2021-03-09T09:45:02.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:02.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:02.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e150 con 0x7f63d8005e00 2021-03-09T09:45:02.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:45:02.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.307+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:02.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.308+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:45:02.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.308+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:02.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.308+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067d20 secure :-1 s=READY pgs=9333 cs=0 l=1 rev1=1 rx=0x7f63e0012b30 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:02.334 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:02.309+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25539 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:02.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d9a0 con 0x7f63d8005e00 2021-03-09T09:45:02.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.310+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:02.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.310+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:02.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.310+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:02.338 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:02.338 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:02.339 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:02.339 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:02.340 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:02.340 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:02.340 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:02.341 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:02.341 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:02.342 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:02.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d9a0 con 0x7f63d8005e00 2021-03-09T09:45:02.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.311+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:02.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.312+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:45:02.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:45:02.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:02.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:02.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:03.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:03.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:03.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:03.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:04.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:04.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:04.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:04.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:04.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:04.997+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:05.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:05.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:05.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:05.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:06.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:06.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:06.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:06.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0067d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:45:07.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0067d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:07.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067d20 secure :-1 s=READY pgs=9333 cs=0 l=1 rev1=1 rx=0x7f63e0012b30 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:07.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067d20 secure :-1 s=READY pgs=9333 cs=0 l=1 rev1=1 rx=0x7f63e0012b30 tx=0x7f63fc04a740).stop 2021-03-09T09:45:07.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0067d20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067d20 unknown :-1 s=CLOSED pgs=9333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:07.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:45:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:07.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:07.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:45:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8008840 2021-03-09T09:45:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:07.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:45:07.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:07.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:07.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:07.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:07.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:45:07.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:07.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:07.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:07.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:07.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.310+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:07.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:07.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:07.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:07.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:07.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:07.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:07.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:07.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:45:07.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:45:07.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:07.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:07.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bd50 con 0x7f6400146050 2021-03-09T09:45:07.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d4a0 con 0x7f6400146050 2021-03-09T09:45:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:45:07.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cd80 secure :-1 s=READY pgs=9345 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc012ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:45:07.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.311+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:07.326 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:07.313+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25579 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:07.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023a20 con 0x7f6400146050 2021-03-09T09:45:07.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.314+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:07.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.314+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:07.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.314+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:07.328 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:07.329 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:07.329 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023e80 con 0x7f6400146050 2021-03-09T09:45:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.314+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.314+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:45:07.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.315+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T09:45:07.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:07.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:08.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:08.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:08.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:08.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:08.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:08.523+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:09.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:09.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:09.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:09.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:10.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:09.998+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:10.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:10.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:10.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:10.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:11.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:11.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:11.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:12.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.313+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:12.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.313+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:12.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.313+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cd80 secure :-1 s=READY pgs=9345 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc012ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:12.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.313+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cd80 secure :-1 s=READY pgs=9345 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc012ec0).stop 2021-03-09T09:45:12.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:12.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:12.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006cd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:12.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006cd80 unknown :-1 s=CLOSED pgs=9345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:12.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:45:12.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:12.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:12.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:12.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:12.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:12.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:12.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:12.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T09:45:12.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:12.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d1b0 con 0x7f63d8008840 2021-03-09T09:45:12.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:12.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065c10 con 0x7f63d8005e00 2021-03-09T09:45:12.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:12.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:12.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:12.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:12.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:12.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:12.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:45:12.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:12.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.314+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:12.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:12.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:12.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:12.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:12.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:12.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:12.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:12.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:12.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:45:12.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:45:12.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:12.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:12.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T09:45:12.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:12.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:12.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bd50 con 0x7f63d8005e00 2021-03-09T09:45:12.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:45:12.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.315+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:12.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.316+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c120 secure :-1 s=READY pgs=9361 cs=0 l=1 rev1=1 rx=0x7f63fc013f60 tx=0x7f63fc04f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:12.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.316+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:45:12.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.316+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:12.329 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:12.317+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25614 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:12.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023e80 con 0x7f63d8005e00 2021-03-09T09:45:12.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.318+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:12.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.318+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:12.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.318+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:12.331 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:12.331 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:12.331 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:12.332 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:12.332 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:12.332 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:12.332 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:12.332 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:12.333 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:12.333 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:12.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023e80 con 0x7f63d8005e00 2021-03-09T09:45:12.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.321+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:12.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8005e00 2021-03-09T09:45:12.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.321+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:45:12.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:12.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:12.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:13.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:13.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:13.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:13.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:14.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:14.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:14.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:14.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:15.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:14.999+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:15.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:15.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:15.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:15.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:16.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:16.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:16.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:16.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:17.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:17.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:17.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c120 secure :-1 s=READY pgs=9361 cs=0 l=1 rev1=1 rx=0x7f63fc013f60 tx=0x7f63fc04f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:17.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c120 secure :-1 s=READY pgs=9361 cs=0 l=1 rev1=1 rx=0x7f63fc013f60 tx=0x7f63fc04f000).stop 2021-03-09T09:45:17.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:17.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:17.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007c120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:17.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007c120 unknown :-1 s=CLOSED pgs=9361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:17.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:45:17.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:17.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:17.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:17.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:17.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:17.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:17.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:17.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ae80 con 0x7f63d8003400 2021-03-09T09:45:17.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:17.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025840 con 0x7f63d8008840 2021-03-09T09:45:17.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:17.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f6400146050 2021-03-09T09:45:17.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:17.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:17.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:17.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:17.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.318+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:17.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.319+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:17.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.319+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:45:17.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.319+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:17.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.319+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:17.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:17.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:17.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:17.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:17.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:17.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:17.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:17.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:17.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:45:17.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:45:17.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:17.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:17.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T09:45:17.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:17.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:17.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.320+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:45:17.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.321+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:45:17.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.321+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:17.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.321+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:45:17.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.321+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:17.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.321+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 secure :-1 s=READY pgs=9377 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc066f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:17.347 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:17.322+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25656 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:17.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.322+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f6400146050 2021-03-09T09:45:17.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.323+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:17.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.323+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:17.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.323+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:17.350 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:17.350 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:17.351 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:17.351 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:17.351 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:17.352 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:17.353 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:17.353 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:17.353 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:17.354 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:17.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04d9a0 con 0x7f6400146050 2021-03-09T09:45:17.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.323+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:17.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.323+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:45:17.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f6400146050 2021-03-09T09:45:17.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:17.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:17.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:18.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:18.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:18.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:18.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:18.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:18.524+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:19.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:19.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:19.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:19.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:20.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:20.000+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:20.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:20.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:20.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:20.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:21.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:21.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:21.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:21.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:22.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:45:22.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:22.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 secure :-1 s=READY pgs=9377 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc066f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:22.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 secure :-1 s=READY pgs=9377 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc066f20).stop 2021-03-09T09:45:22.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:22.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:22.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e006c5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:22.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e006c5b0 unknown :-1 s=CLOSED pgs=9377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:22.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:45:22.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:22.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:22.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:22.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:22.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:22.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:22.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:22.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:45:22.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:22.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T09:45:22.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:22.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bc00 con 0x7f63d8005e00 2021-03-09T09:45:22.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:22.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:22.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:22.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:22.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:22.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:22.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:45:22.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:22.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.323+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:22.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:22.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:22.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:22.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:22.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:22.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:22.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:22.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:22.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:45:22.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:45:22.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:22.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:22.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:45:22.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:22.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:22.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T09:45:22.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:45:22.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.324+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:22.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.325+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d4f0 secure :-1 s=READY pgs=9388 cs=0 l=1 rev1=1 rx=0x7f63e005dc10 tx=0x7f63fc04f000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:22.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.325+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:45:22.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.325+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:22.340 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:22.326+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25693 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:22.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052440 con 0x7f63d8005e00 2021-03-09T09:45:22.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.327+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:22.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.327+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:22.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.327+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:22.341 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:22.341 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:22.341 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:22.342 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:22.342 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:22.342 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:22.342 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:22.343 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:22.343 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:22.343 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:22.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63d8005e00 2021-03-09T09:45:22.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.327+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:22.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.327+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:45:22.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:45:22.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:22.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:22.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:23.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:23.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:23.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:24.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:24.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:24.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:24.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:25.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:25.001+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:25.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:25.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:25.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:25.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:26.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:26.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:26.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:26.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:27.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:27.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:27.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d4f0 secure :-1 s=READY pgs=9388 cs=0 l=1 rev1=1 rx=0x7f63e005dc10 tx=0x7f63fc04f000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:27.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d4f0 secure :-1 s=READY pgs=9388 cs=0 l=1 rev1=1 rx=0x7f63e005dc10 tx=0x7f63fc04f000).stop 2021-03-09T09:45:27.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:27.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:27.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d4f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:27.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d4f0 unknown :-1 s=CLOSED pgs=9388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:27.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:45:27.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:27.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:27.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:27.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:27.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:27.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:27.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:27.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065c10 con 0x7f63d8003400 2021-03-09T09:45:27.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:27.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:45:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f6400146050 2021-03-09T09:45:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:27.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:27.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:45:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.327+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:27.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:27.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:27.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:27.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:45:27.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:45:27.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:27.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:27.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T09:45:27.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:27.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:27.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f6400146050 2021-03-09T09:45:27.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:45:27.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:45:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cc10 secure :-1 s=READY pgs=9404 cs=0 l=1 rev1=1 rx=0x7f63f000b040 tx=0x7f63f0003ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:27.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.329+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:27.343 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:27.330+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25733 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:27.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f6400146050 2021-03-09T09:45:27.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.331+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:27.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.331+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:27.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.331+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:27.345 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:27.345 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:27.345 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:27.345 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:27.345 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:27.346 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:27.346 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:27.346 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:27.346 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:27.347 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:27.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:45:27.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.333+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:27.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.333+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:45:27.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00036b0 con 0x7f6400146050 2021-03-09T09:45:27.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:27.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:27.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:28.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:28.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:28.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:28.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:28.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:28.525+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:29.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:29.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:29.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:29.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:30.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:30.002+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:30.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:30.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:30.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:30.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:31.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:31.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:31.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:32.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:45:32.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:32.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cc10 secure :-1 s=READY pgs=9404 cs=0 l=1 rev1=1 rx=0x7f63f000b040 tx=0x7f63f0003ac0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:32.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cc10 secure :-1 s=READY pgs=9404 cs=0 l=1 rev1=1 rx=0x7f63f000b040 tx=0x7f63f0003ac0).stop 2021-03-09T09:45:32.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:32.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:32.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e001cc10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:32.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e001cc10 unknown :-1 s=CLOSED pgs=9404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:32.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:45:32.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:32.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:32.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:32.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:32.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:32.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:32.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:32.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63d8003400 2021-03-09T09:45:32.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:32.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e730 con 0x7f63d8008840 2021-03-09T09:45:32.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:32.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8005e00 2021-03-09T09:45:32.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:32.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:32.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:32.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:32.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:32.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:32.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:45:32.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:32.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.331+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:32.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:32.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:32.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:32.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:32.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:32.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:32.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:32.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:32.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:45:32.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:45:32.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:32.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:32.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014060 con 0x7f63d8005e00 2021-03-09T09:45:32.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:32.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:32.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:45:32.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:45:32.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.332+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:32.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.334+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:45:32.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.334+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:32.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.334+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bfa0 secure :-1 s=READY pgs=9418 cs=0 l=1 rev1=1 rx=0x7f63f404d630 tx=0x7f63f40099f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:32.349 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:32.334+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25770 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:32.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404d290 con 0x7f63d8005e00 2021-03-09T09:45:32.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.335+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:32.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.335+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:32.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.335+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:32.351 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:32.351 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:32.351 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:32.352 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:32.352 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:32.352 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:32.352 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:32.353 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:32.353 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:32.353 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:32.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.336+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404d290 con 0x7f63d8005e00 2021-03-09T09:45:32.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.336+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:32.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.336+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:45:32.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.337+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f63d8005e00 2021-03-09T09:45:32.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:32.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:32.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:33.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:33.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:33.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:33.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:34.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:34.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:34.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:34.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:35.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:35.003+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:35.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:35.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:35.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:35.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:36.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:36.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:37.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:37.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:37.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bfa0 secure :-1 s=READY pgs=9418 cs=0 l=1 rev1=1 rx=0x7f63f404d630 tx=0x7f63f40099f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:37.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bfa0 secure :-1 s=READY pgs=9418 cs=0 l=1 rev1=1 rx=0x7f63f404d630 tx=0x7f63f40099f0).stop 2021-03-09T09:45:37.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:37.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:37.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007bfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:37.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bfa0 unknown :-1 s=CLOSED pgs=9418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:37.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:45:37.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:37.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:37.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:37.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:37.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:37.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:37.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:37.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bc00 con 0x7f63d8003400 2021-03-09T09:45:37.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:37.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:45:37.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:37.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f4d0 con 0x7f6400146050 2021-03-09T09:45:37.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.334+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:37.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.335+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:37.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.335+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:37.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:37.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.335+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:37.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.335+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:37.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.336+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:45:37.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.336+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:37.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.336+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:37.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.336+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:37.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.336+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:37.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.336+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:37.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:37.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:37.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:37.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:37.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:37.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:45:37.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:45:37.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:37.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:37.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f6400146050 2021-03-09T09:45:37.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:37.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:37.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T09:45:37.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:45:37.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.337+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:37.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.338+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:45:37.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.338+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015780 secure :-1 s=READY pgs=9433 cs=0 l=1 rev1=1 rx=0x7f63f0066070 tx=0x7f63f0028f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:37.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.338+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:37.351 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:37.338+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25809 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:37.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.339+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f6400146050 2021-03-09T09:45:37.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.339+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:37.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.339+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:37.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.339+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:37.353 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:37.353 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:37.354 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:37.354 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:37.354 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:37.355 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:37.355 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:37.355 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:37.355 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:37.355 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:37.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.342+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:45:37.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.342+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:37.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.342+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:45:37.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:45:37.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:37.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:37.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:38.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:38.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:38.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:38.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:38.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:38.527+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:39.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:39.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:39.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:39.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:40.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:40.004+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:40.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:40.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:40.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:40.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:41.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:41.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:41.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:41.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:42.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:42.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:42.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015780 secure :-1 s=READY pgs=9433 cs=0 l=1 rev1=1 rx=0x7f63f0066070 tx=0x7f63f0028f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:42.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015780 secure :-1 s=READY pgs=9433 cs=0 l=1 rev1=1 rx=0x7f63f0066070 tx=0x7f63f0028f00).stop 2021-03-09T09:45:42.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:42.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:42.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0015780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:42.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0015780 unknown :-1 s=CLOSED pgs=9433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:42.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:45:42.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:42.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:42.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:42.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:42.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:42.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:42.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:42.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010940 con 0x7f63d8003400 2021-03-09T09:45:42.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:42.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8008840 2021-03-09T09:45:42.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:42.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065c10 con 0x7f63d8005e00 2021-03-09T09:45:42.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:42.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:42.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.338+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:42.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.339+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:42.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.339+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:42.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.339+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:42.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.339+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:45:42.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.339+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:42.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.339+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:42.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.339+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:42.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.339+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:42.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.339+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:42.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:42.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:42.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:42.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:42.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:42.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:45:42.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:45:42.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:42.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:42.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00680e0 con 0x7f63d8005e00 2021-03-09T09:45:42.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:42.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:42.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014060 con 0x7f63d8005e00 2021-03-09T09:45:42.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:45:42.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.340+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:42.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.341+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:45:42.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.341+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002c7d0 secure :-1 s=READY pgs=9447 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f40035e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:42.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.341+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:42.355 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:42.342+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25848 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:42.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4061c00 con 0x7f63d8005e00 2021-03-09T09:45:42.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.343+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:42.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.343+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:42.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.343+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:42.357 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:42.357 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:42.358 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:42.358 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:42.358 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:42.358 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:42.358 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:42.359 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:42.359 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:42.359 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:42.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4061c00 con 0x7f63d8005e00 2021-03-09T09:45:42.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.344+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:42.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.344+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:45:42.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8005e00 2021-03-09T09:45:42.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:42.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:42.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:43.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:43.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:43.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:43.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:44.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:44.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:44.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:44.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:45.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:45.005+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:45.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:45.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:45.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:45.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:46.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:46.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:46.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:46.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:47.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e002c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:45:47.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e002c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:47.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002c7d0 secure :-1 s=READY pgs=9447 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f40035e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:47.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002c7d0 secure :-1 s=READY pgs=9447 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f40035e0).stop 2021-03-09T09:45:47.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:47.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:47.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e002c7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:47.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002c7d0 unknown :-1 s=CLOSED pgs=9447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:47.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:45:47.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:47.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:47.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:47.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:47.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:47.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:47.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:47.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:45:47.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:47.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00199c0 con 0x7f63d8008840 2021-03-09T09:45:47.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:47.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017bf0 con 0x7f6400146050 2021-03-09T09:45:47.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:47.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:47.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:47.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:47.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:47.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:47.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:45:47.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:47.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.343+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:47.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:47.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:47.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:47.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:47.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:47.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:47.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:47.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:47.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:45:47.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:45:47.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:47.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:47.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:45:47.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:47.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:47.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:45:47.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:45:47.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.344+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:47.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.345+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e1f0 secure :-1 s=READY pgs=9459 cs=0 l=1 rev1=1 rx=0x7f63fc075fb0 tx=0x7f63fc05ef20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:47.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.345+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:45:47.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.345+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:47.358 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:47.346+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25888 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:47.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T09:45:47.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.348+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:47.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.348+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:47.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.348+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:47.360 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:47.360 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:47.360 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:47.360 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:47.361 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:47.361 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:47.361 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:47.361 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:47.361 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:47.362 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:47.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f6400146050 2021-03-09T09:45:47.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.349+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:47.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.349+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:45:47.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.350+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc05ebb0 con 0x7f6400146050 2021-03-09T09:45:47.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:47.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:47.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:48.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:48.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:48.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:48.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:48.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:48.528+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:49.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:49.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:50.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:50.006+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:50.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:50.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:50.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:50.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:51.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:51.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:51.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:51.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:52.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:52.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:52.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e1f0 secure :-1 s=READY pgs=9459 cs=0 l=1 rev1=1 rx=0x7f63fc075fb0 tx=0x7f63fc05ef20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:52.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e1f0 secure :-1 s=READY pgs=9459 cs=0 l=1 rev1=1 rx=0x7f63fc075fb0 tx=0x7f63fc05ef20).stop 2021-03-09T09:45:52.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e002e1f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e002e1f0 unknown :-1 s=CLOSED pgs=9459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:45:52.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:52.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:45:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8008840 2021-03-09T09:45:52.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:52.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008a40 con 0x7f63d8005e00 2021-03-09T09:45:52.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:52.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:52.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.347+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:52.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.348+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:52.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.348+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:52.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:52.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.349+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:45:52.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.349+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:52.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.349+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.350+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.350+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:52.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.350+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:52.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:52.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:52.359 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:52.350+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25925 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:52.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:52.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:45:52.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:45:52.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:52.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:52.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:45:52.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:52.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:52.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:45:52.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:45:52.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:52.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bca0 secure :-1 s=READY pgs=9480 cs=0 l=1 rev1=1 rx=0x7f63f00159f0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:52.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:45:52.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:52.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000eef0 con 0x7f63d8005e00 2021-03-09T09:45:52.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:52.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:52.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.351+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:52.364 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:52.365 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:52.365 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:52.365 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:52.365 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:52.366 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:52.366 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:52.366 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:52.366 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:52.366 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:52.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000eef0 con 0x7f63d8005e00 2021-03-09T09:45:52.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.352+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:52.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.352+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:45:52.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.353+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T09:45:52.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:52.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:52.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:53.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:53.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:53.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:53.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:54.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:54.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:54.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:54.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:55.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:55.007+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:45:55.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:55.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:55.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:55.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:56.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:56.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:56.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:56.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:57.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:45:57.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:45:57.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bca0 secure :-1 s=READY pgs=9480 cs=0 l=1 rev1=1 rx=0x7f63f00159f0 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:45:57.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bca0 secure :-1 s=READY pgs=9480 cs=0 l=1 rev1=1 rx=0x7f63f00159f0 tx=0x7f63f004db20).stop 2021-03-09T09:45:57.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:57.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:45:57.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007bca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:45:57.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007bca0 unknown :-1 s=CLOSED pgs=9480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:45:57.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:45:57.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:57.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:45:57.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:57.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:45:57.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:45:57.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:45:57.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:57.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065c10 con 0x7f63d8003400 2021-03-09T09:45:57.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:57.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00262a0 con 0x7f63d8008840 2021-03-09T09:45:57.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:45:57.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:45:57.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.351+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:45:57.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.352+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:45:57.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.352+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:57.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.352+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:45:57.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.352+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:45:57.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:45:57.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.352+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:45:57.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.352+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:45:57.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.352+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:45:57.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.353+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:45:57.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.353+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:45:57.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.353+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:45:57.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:45:57.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:57.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:57.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:45:57.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:45:57.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:45:57.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:45:57.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:57.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:57.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a490 con 0x7f6400146050 2021-03-09T09:45:57.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:45:57.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:45:57.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:45:57.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:45:57.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:57.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005df80 secure :-1 s=READY pgs=9490 cs=0 l=1 rev1=1 rx=0x7f63f005dcd0 tx=0x7f63f0009580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:45:57.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:45:57.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:45:57.368 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:45:57.354+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 25965 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:45:57.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000eef0 con 0x7f6400146050 2021-03-09T09:45:57.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.355+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:45:57.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.355+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:45:57.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.355+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:45:57.369 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:45:57.370 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:45:57.370 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:57.370 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:45:57.370 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:45:57.371 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:45:57.371 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:45:57.371 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:45:57.371 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:45:57.371 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:45:57.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000eef0 con 0x7f6400146050 2021-03-09T09:45:57.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.356+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:45:57.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.356+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:45:57.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.357+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038470 con 0x7f6400146050 2021-03-09T09:45:57.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:57.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:57.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:58.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:58.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:58.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:58.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:45:58.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:58.529+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:45:59.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:59.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:45:59.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:45:59.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:00.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:00.008+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:00.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:00.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:00.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:00.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:01.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:01.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:01.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:01.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:02.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:46:02.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:02.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005df80 secure :-1 s=READY pgs=9490 cs=0 l=1 rev1=1 rx=0x7f63f005dcd0 tx=0x7f63f0009580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:02.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005df80 secure :-1 s=READY pgs=9490 cs=0 l=1 rev1=1 rx=0x7f63f005dcd0 tx=0x7f63f0009580).stop 2021-03-09T09:46:02.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:02.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:02.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e005df80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:02.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e005df80 unknown :-1 s=CLOSED pgs=9490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:02.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:46:02.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:02.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:02.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:02.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:02.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:02.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:02.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:02.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e160 con 0x7f63d8003400 2021-03-09T09:46:02.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:02.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:46:02.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:02.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de80 con 0x7f63d8005e00 2021-03-09T09:46:02.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:02.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:02.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.356+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:02.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.357+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:02.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.357+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:46:02.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.357+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:02.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.357+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:02.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.357+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:02.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.357+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:02.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.357+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:02.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.357+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:02.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.357+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:02.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:02.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:02.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:02.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:02.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:02.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:46:02.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:46:02.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:02.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:02.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027af0 con 0x7f63d8005e00 2021-03-09T09:46:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:46:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:46:02.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.358+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:02.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.359+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:46:02.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.359+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:02.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.359+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007ebc0 secure :-1 s=READY pgs=9501 cs=0 l=1 rev1=1 rx=0x7f63e0012b00 tx=0x7f63fc04a7b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:02.371 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:02.359+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26012 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:02.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:46:02.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.361+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:02.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.361+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:02.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.361+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:02.373 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:02.373 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:02.373 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:02.373 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:02.373 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:02.374 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:02.374 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.363+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8005e00 2021-03-09T09:46:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.363+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.364+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:46:02.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.366+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:46:02.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:02.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:02.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:03.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:03.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:03.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:03.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:04.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:04.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:04.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:04.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:05.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:05.009+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:05.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:05.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:05.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:05.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:06.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:06.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:06.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:06.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:07.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:07.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:07.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007ebc0 secure :-1 s=READY pgs=9501 cs=0 l=1 rev1=1 rx=0x7f63e0012b00 tx=0x7f63fc04a7b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:07.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007ebc0 secure :-1 s=READY pgs=9501 cs=0 l=1 rev1=1 rx=0x7f63e0012b00 tx=0x7f63fc04a7b0).stop 2021-03-09T09:46:07.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:07.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:07.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007ebc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:07.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007ebc0 unknown :-1 s=CLOSED pgs=9501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:07.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:46:07.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:07.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:07.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:07.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:07.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:07.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:07.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:07.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008a40 con 0x7f63d8003400 2021-03-09T09:46:07.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:07.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8008840 2021-03-09T09:46:07.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:07.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068d20 con 0x7f6400146050 2021-03-09T09:46:07.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:07.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:07.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:07.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:07.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.360+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:07.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.361+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:07.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.361+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:46:07.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.361+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:07.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.361+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:07.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.362+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:07.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.362+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:07.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.362+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:07.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:07.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:07.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:07.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:07.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:07.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:46:07.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:46:07.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:07.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:07.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T09:46:07.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:07.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:07.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027af0 con 0x7f6400146050 2021-03-09T09:46:07.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:46:07.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:07.373 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:07.363+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26052 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:07.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.364+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9517 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc04fb40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:07.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.364+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:46:07.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.364+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:07.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f6400146050 2021-03-09T09:46:07.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.368+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:07.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.368+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:07.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.368+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:07.376 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:07.376 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:07.376 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:07.376 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:07.377 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:07.377 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:07.377 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:07.377 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:07.377 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:07.378 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:07.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.370+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f6400146050 2021-03-09T09:46:07.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.370+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:07.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.370+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:46:07.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02dc10 con 0x7f6400146050 2021-03-09T09:46:07.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:07.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:07.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:08.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:08.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:08.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:08.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:08.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:08.531+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:09.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:09.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:09.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:09.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:10.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:10.010+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:10.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:10.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:10.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:10.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:11.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:11.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:11.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:11.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:12.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:12.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9517 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc04fb40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9517 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc04fb40).stop 2021-03-09T09:46:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:46:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012790 con 0x7f63d8003400 2021-03-09T09:46:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065c10 con 0x7f63d8008840 2021-03-09T09:46:12.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:12.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T09:46:12.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:12.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:12.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.364+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:12.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.365+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:12.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.365+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:12.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.365+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:12.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.366+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:46:12.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.366+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:12.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.366+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:12.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:12.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:12.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:12.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:12.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:46:12.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:46:12.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:12.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:12.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:46:12.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:12.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:12.378 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:12.368+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26089 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:12.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.367+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T09:46:12.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.368+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:46:12.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.368+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:12.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.368+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=9535 cs=0 l=1 rev1=1 rx=0x7f63e00163a0 tx=0x7f63fc04fd00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:12.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.368+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:46:12.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.368+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:12.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.370+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T09:46:12.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.370+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:12.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.370+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:12.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.370+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:12.382 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:12.382 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:12.382 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:12.382 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:12.383 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:12.383 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:12.383 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:12.383 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:12.384 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:12.384 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:12.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T09:46:12.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.372+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:12.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.372+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:46:12.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.373+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T09:46:12.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:12.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:12.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:13.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:13.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:13.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:13.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:14.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:14.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:14.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:14.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:15.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:15.011+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:15.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:15.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:15.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:15.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:16.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:16.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:16.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:16.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:17.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.368+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:17.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.368+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:17.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.368+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=9535 cs=0 l=1 rev1=1 rx=0x7f63e00163a0 tx=0x7f63fc04fd00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:17.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.368+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=9535 cs=0 l=1 rev1=1 rx=0x7f63e00163a0 tx=0x7f63fc04fd00).stop 2021-03-09T09:46:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=CLOSED pgs=9535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:46:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:17.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:17.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:17.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:17.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:17.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63d8003400 2021-03-09T09:46:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8008840 2021-03-09T09:46:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a260 con 0x7f6400146050 2021-03-09T09:46:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:17.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:17.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:17.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:17.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.369+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:17.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.370+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:17.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.370+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:46:17.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.370+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:17.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.370+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:17.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.370+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:17.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.370+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:17.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.371+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:17.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:17.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:17.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:17.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:17.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:17.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:46:17.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:46:17.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:17.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:17.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:46:17.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:17.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:17.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00680e0 con 0x7f6400146050 2021-03-09T09:46:17.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:46:17.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:17.382 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26129 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:17.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9546 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:17.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:46:17.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.372+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:17.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022570 con 0x7f6400146050 2021-03-09T09:46:17.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.374+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:17.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.374+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:17.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.374+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:17.385 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:17.385 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:17.385 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:17.386 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:17.386 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:17.386 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:17.386 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:17.387 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:17.387 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:17.387 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:17.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:46:17.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.375+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:17.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.375+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:46:17.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f6400146050 2021-03-09T09:46:17.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:17.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:18.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:18.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:18.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:18.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:18.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:18.532+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:19.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:19.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:19.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:19.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:20.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:20.012+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:20.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:20.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:20.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:20.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:21.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:21.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:21.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:21.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:22.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:22.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:22.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9546 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:22.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9546 cs=0 l=1 rev1=1 rx=0x7f63f4009c90 tx=0x7f63f4017ed0).stop 2021-03-09T09:46:22.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:22.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:22.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:22.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:22.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:46:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068d20 con 0x7f63d8003400 2021-03-09T09:46:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cfe0 con 0x7f63d8008840 2021-03-09T09:46:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:46:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.372+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.373+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:22.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.373+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:22.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.373+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:22.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.373+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:22.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.373+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:22.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.373+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:46:22.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.373+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:22.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.373+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:22.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:46:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:46:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:46:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a490 con 0x7f63d8005e00 2021-03-09T09:46:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:46:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.374+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.375+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:46:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.375+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:22.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.375+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=9555 cs=0 l=1 rev1=1 rx=0x7f63f005f8b0 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:22.387 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:22.376+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26166 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T09:46:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.377+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:22.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.377+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:22.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.377+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:22.389 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:22.389 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:22.389 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:22.390 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:22.390 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:22.390 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:22.390 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:22.390 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:22.391 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:22.391 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:22.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T09:46:22.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.380+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:22.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.380+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:46:22.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.381+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00412d0 con 0x7f63d8005e00 2021-03-09T09:46:22.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:22.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:22.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:23.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:23.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:23.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:23.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:24.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:24.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:24.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:24.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:25.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:25.013+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:25.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:25.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:25.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:25.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:26.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:26.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:26.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:26.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:27.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:27.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=9555 cs=0 l=1 rev1=1 rx=0x7f63f005f8b0 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=9555 cs=0 l=1 rev1=1 rx=0x7f63f005f8b0 tx=0x7f63f0000bc0).stop 2021-03-09T09:46:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:27.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=CLOSED pgs=9555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:27.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:46:27.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:27.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:27.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:46:27.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:27.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f860 con 0x7f63d8008840 2021-03-09T09:46:27.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:27.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067670 con 0x7f6400146050 2021-03-09T09:46:27.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.375+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:27.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.376+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:27.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.376+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:27.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.376+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:27.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.376+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.376+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.376+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:46:27.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.376+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:27.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.376+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:27.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:27.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:27.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:27.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:27.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:27.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:27.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:27.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:27.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:46:27.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:46:27.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:27.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:27.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005df80 con 0x7f6400146050 2021-03-09T09:46:27.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:27.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:27.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:46:27.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:46:27.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.377+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:27.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.378+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:46:27.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.378+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9569 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f401ae60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:27.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.378+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:27.398 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:27.379+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26206 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:27.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.381+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4018340 con 0x7f6400146050 2021-03-09T09:46:27.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.381+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:27.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.381+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:27.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.381+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:27.401 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:27.401 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:27.402 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:27.402 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:27.402 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:27.403 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:27.403 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:27.404 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:27.404 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:27.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.384+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034ba0 con 0x7f6400146050 2021-03-09T09:46:27.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.384+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.384+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:46:27.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.384+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4018340 con 0x7f6400146050 2021-03-09T09:46:27.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:27.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:27.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:28.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:28.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:28.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:28.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:28.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:28.532+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:29.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:29.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:29.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:29.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:30.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:30.013+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:30.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:30.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:30.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:30.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:31.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:31.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:31.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:31.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:32.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:32.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:32.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9569 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f401ae60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:32.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9569 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f401ae60).stop 2021-03-09T09:46:32.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:32.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:32.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:32.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:32.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:46:32.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:32.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:32.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0065690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:32.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:32.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:32.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:32.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:32.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63d8003400 2021-03-09T09:46:32.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:32.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63d8008840 2021-03-09T09:46:32.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:32.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:46:32.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.379+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:32.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.380+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:32.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.380+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:32.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.380+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:32.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.380+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:32.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0065690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.380+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:46:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.380+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.380+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.381+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.381+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.381+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.381+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:32.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:32.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:32.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:32.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:46:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:46:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f63d8005e00 2021-03-09T09:46:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:46:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:46:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:32.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:46:32.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:32.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.382+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0065690 secure :-1 s=READY pgs=9584 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc013f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:32.395 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:32.383+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26243 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:32.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:46:32.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.385+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:32.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.385+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:32.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.385+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:32.397 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:32.397 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:32.398 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:32.398 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:32.398 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:32.398 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:32.398 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:32.399 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:32.399 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:32.399 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:32.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8005e00 2021-03-09T09:46:32.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.387+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:32.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.387+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:46:32.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.389+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:46:32.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:32.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:32.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:33.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:33.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:33.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:33.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:34.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:34.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:34.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:34.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:35.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:35.014+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:35.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:35.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:35.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:35.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:36.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:36.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:37.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0065690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:37.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0065690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:37.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0065690 secure :-1 s=READY pgs=9584 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc013f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:37.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0065690 secure :-1 s=READY pgs=9584 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc013f20).stop 2021-03-09T09:46:37.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:37.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:37.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0065690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:37.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0065690 unknown :-1 s=CLOSED pgs=9584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:37.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:46:37.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:37.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:37.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:37.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:37.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:37.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:37.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:37.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001f860 con 0x7f63d8003400 2021-03-09T09:46:37.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:37.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001dab0 con 0x7f63d8008840 2021-03-09T09:46:37.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:37.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023e00 con 0x7f6400146050 2021-03-09T09:46:37.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:37.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:37.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.384+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:37.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.385+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:37.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.385+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:46:37.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.385+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:37.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.385+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:37.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.385+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:37.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.385+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:37.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.386+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:37.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.386+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:37.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.386+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:37.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.386+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:46:37.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:46:37.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:37.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:37.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f6400146050 2021-03-09T09:46:37.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:37.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:37.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:46:37.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:46:37.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:37.399 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:37.387+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26283 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:37.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.388+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9600 cs=0 l=1 rev1=1 rx=0x7f63f007efd0 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:37.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.388+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:46:37.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.388+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:37.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.390+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0003d30 con 0x7f6400146050 2021-03-09T09:46:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.390+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.390+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.390+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:37.402 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:37.402 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:37.402 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:37.402 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:37.403 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:37.403 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:37.403 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:37.403 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:37.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0003d30 con 0x7f6400146050 2021-03-09T09:46:37.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.392+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:37.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.392+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:46:37.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f006dd90 con 0x7f6400146050 2021-03-09T09:46:37.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:37.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:37.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:38.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:38.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:38.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:38.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:38.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:38.533+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:39.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:39.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:39.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:39.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:40.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:40.015+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:40.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:40.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:40.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:41.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:41.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:41.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:42.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:42.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:42.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9600 cs=0 l=1 rev1=1 rx=0x7f63f007efd0 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:42.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9600 cs=0 l=1 rev1=1 rx=0x7f63f007efd0 tx=0x7f63f0013b80).stop 2021-03-09T09:46:42.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:42.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:42.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:42.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9600 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:42.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:46:42.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:42.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:42.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:42.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:42.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:42.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:42.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:42.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:46:42.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:42.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63d8008840 2021-03-09T09:46:42.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:42.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016650 con 0x7f63d8005e00 2021-03-09T09:46:42.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:42.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:42.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:42.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:42.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.388+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:42.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.389+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:42.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.389+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:46:42.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.389+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:42.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.389+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:42.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.389+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:42.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.389+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:42.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.389+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:42.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:42.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:42.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:42.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:42.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:42.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:46:42.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:46:42.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:42.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:42.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d30 con 0x7f63d8005e00 2021-03-09T09:46:42.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:42.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:42.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f63d8005e00 2021-03-09T09:46:42.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:46:42.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:42.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000d0b0 secure :-1 s=READY pgs=9609 cs=0 l=1 rev1=1 rx=0x7f63f0088fe0 tx=0x7f63f0088e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:42.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:46:42.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.390+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:42.403 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:42.391+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26320 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:42.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0003d30 con 0x7f63d8005e00 2021-03-09T09:46:42.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.392+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:42.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.392+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.392+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:42.406 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:42.406 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:42.406 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:42.406 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:42.407 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:42.407 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:42.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0003d30 con 0x7f63d8005e00 2021-03-09T09:46:42.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.393+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:42.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.393+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:46:42.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.394+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00298c0 con 0x7f63d8005e00 2021-03-09T09:46:42.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:42.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:42.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:43.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:43.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:43.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:43.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:44.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:44.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:44.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:44.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:45.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:45.016+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:45.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:45.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:45.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:45.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:46.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:46.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:46.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:46.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:47.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:47.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:47.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000d0b0 secure :-1 s=READY pgs=9609 cs=0 l=1 rev1=1 rx=0x7f63f0088fe0 tx=0x7f63f0088e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:47.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000d0b0 secure :-1 s=READY pgs=9609 cs=0 l=1 rev1=1 rx=0x7f63f0088fe0 tx=0x7f63f0088e20).stop 2021-03-09T09:46:47.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:47.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:47.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000d0b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:47.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000d0b0 unknown :-1 s=CLOSED pgs=9609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:47.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:46:47.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:47.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:47.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:47.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:47.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:47.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:47.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:47.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026010 con 0x7f63d8003400 2021-03-09T09:46:47.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:47.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00678b0 con 0x7f63d8008840 2021-03-09T09:46:47.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:47.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010080 con 0x7f6400146050 2021-03-09T09:46:47.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:47.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:47.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:47.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:47.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.392+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:47.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:47.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:46:47.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:47.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:47.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:47.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:47.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:47.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:47.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:47.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:47.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:47.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.393+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:47.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:46:47.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:46:47.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:47.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:47.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:46:47.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:47.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:47.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005df80 con 0x7f6400146050 2021-03-09T09:46:47.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:46:47.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:47.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9622 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4070170).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:47.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:46:47.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.394+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:47.413 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:47.396+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26360 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:47.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f6400146050 2021-03-09T09:46:47.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.396+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:47.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.396+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:47.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.396+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:47.416 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:47.416 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:47.416 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:47.416 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:47.416 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f6400146050 2021-03-09T09:46:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.396+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.396+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:46:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f6400146050 2021-03-09T09:46:47.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:47.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:47.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:48.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:48.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:48.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:48.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:48.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:48.535+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:49.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:49.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:49.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:49.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:50.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:50.017+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:50.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:50.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:50.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:50.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:51.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:51.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:51.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:51.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:52.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:46:52.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:52.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9622 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4070170).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:52.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9622 cs=0 l=1 rev1=1 rx=0x7f63f4005950 tx=0x7f63f4070170).stop 2021-03-09T09:46:52.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:52.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:52.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:52.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:52.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:46:52.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:52.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:52.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:52.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:52.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:52.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:52.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:52.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023e00 con 0x7f63d8003400 2021-03-09T09:46:52.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:52.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017380 con 0x7f63d8008840 2021-03-09T09:46:52.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:52.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T09:46:52.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:52.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:52.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:52.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:52.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.396+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:52.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.397+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:52.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.397+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:46:52.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.397+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:52.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.397+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:52.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:52.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:52.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:52.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:52.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:52.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:52.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:52.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:52.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:46:52.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:46:52.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.399+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:52.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:52.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065bc0 con 0x7f63d8005e00 2021-03-09T09:46:52.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:52.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:52.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:46:52.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:46:52.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:52.411 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26397 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001f0a0 secure :-1 s=READY pgs=9644 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f404be70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:52.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:46:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.400+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f406abf0 con 0x7f63d8005e00 2021-03-09T09:46:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.403+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.403+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:52.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.403+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:52.414 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:52.415 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:52.416 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f406ad50 con 0x7f63d8005e00 2021-03-09T09:46:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.403+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.403+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:46:52.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.404+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022870 con 0x7f63d8005e00 2021-03-09T09:46:52.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:52.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:52.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:53.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:53.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:53.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:53.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:54.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:54.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:54.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:54.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:55.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:55.018+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:46:55.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:55.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:55.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:55.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:56.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:56.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:56.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:56.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:57.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.400+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:46:57.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.400+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:46:57.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.400+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001f0a0 secure :-1 s=READY pgs=9644 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f404be70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:46:57.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.400+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001f0a0 secure :-1 s=READY pgs=9644 cs=0 l=1 rev1=1 rx=0x7f63f4005130 tx=0x7f63f404be70).stop 2021-03-09T09:46:57.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:57.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:46:57.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001f0a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:46:57.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001f0a0 unknown :-1 s=CLOSED pgs=9644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:46:57.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:46:57.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:57.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:46:57.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:57.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:46:57.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:46:57.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:46:57.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:57.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002d80 con 0x7f63d8003400 2021-03-09T09:46:57.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:57.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63d8008840 2021-03-09T09:46:57.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:46:57.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aee0 con 0x7f6400146050 2021-03-09T09:46:57.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:46:57.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:46:57.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:57.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:46:57.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.401+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:46:57.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.402+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:46:57.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.402+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:46:57.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.402+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:46:57.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.402+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:46:57.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.402+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:46:57.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.402+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:46:57.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.402+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:46:57.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:46:57.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:57.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:57.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:46:57.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:46:57.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:46:57.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:46:57.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:57.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:57.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a490 con 0x7f6400146050 2021-03-09T09:46:57.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:46:57.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:46:57.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f6400146050 2021-03-09T09:46:57.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.403+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:46:57.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.404+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:57.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.404+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:46:57.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.404+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:46:57.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.404+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9658 cs=0 l=1 rev1=1 rx=0x7f63fc003a80 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:46:57.422 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:46:57.404+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26437 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:46:57.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.404+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f6400146050 2021-03-09T09:46:57.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.404+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:46:57.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.404+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:46:57.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.404+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:46:57.425 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:46:57.425 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:46:57.425 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:57.425 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:46:57.426 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:46:57.426 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:46:57.426 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:46:57.427 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:46:57.427 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:46:57.427 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:46:57.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.407+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f6400146050 2021-03-09T09:46:57.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.407+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:46:57.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.407+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:46:57.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.408+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f6400146050 2021-03-09T09:46:57.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:57.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:57.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:58.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:58.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:58.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:58.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:46:58.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:58.536+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:46:59.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:59.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:46:59.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:46:59.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:00.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:00.019+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:00.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:00.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:00.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:00.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:01.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:01.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:01.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:01.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:02.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.404+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:02.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.404+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:02.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.404+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9658 cs=0 l=1 rev1=1 rx=0x7f63fc003a80 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:02.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.404+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9658 cs=0 l=1 rev1=1 rx=0x7f63fc003a80 tx=0x7f63fc04a6d0).stop 2021-03-09T09:47:02.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.404+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:02.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.404+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:02.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.404+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:02.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:02.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:47:02.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:02.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:02.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:02.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:02.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:02.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:02.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:02.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8003400 2021-03-09T09:47:02.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:02.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023b10 con 0x7f63d8008840 2021-03-09T09:47:02.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:02.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00678b0 con 0x7f63d8005e00 2021-03-09T09:47:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:02.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:47:02.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:02.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.405+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:02.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:02.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:02.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:02.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:02.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:02.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:02.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:02.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:02.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:47:02.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:47:02.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:02.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:02.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000bc00 con 0x7f63d8005e00 2021-03-09T09:47:02.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:02.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:02.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d30 con 0x7f63d8005e00 2021-03-09T09:47:02.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:47:02.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.406+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:02.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.407+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:47:02.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.407+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 secure :-1 s=READY pgs=9663 cs=0 l=1 rev1=1 rx=0x7f63f00426c0 tx=0x7f63f005e9b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:02.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.407+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:02.420 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:02.408+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26475 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:02.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8005e00 2021-03-09T09:47:02.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.409+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:02.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.409+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:02.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.409+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:02.423 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:02.423 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:02.423 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:02.423 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:02.423 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:02.424 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:02.424 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:02.424 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:02.424 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:02.425 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:02.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0042320 con 0x7f63d8005e00 2021-03-09T09:47:02.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.409+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:02.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.409+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:47:02.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.410+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00036b0 con 0x7f63d8005e00 2021-03-09T09:47:02.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:02.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:02.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:03.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:03.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:03.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:03.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:04.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:04.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:04.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:04.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:05.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:05.020+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:05.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:05.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:05.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:05.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:06.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:06.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:06.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:07.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.408+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0018770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:07.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0018770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:07.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 secure :-1 s=READY pgs=9663 cs=0 l=1 rev1=1 rx=0x7f63f00426c0 tx=0x7f63f005e9b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:07.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 secure :-1 s=READY pgs=9663 cs=0 l=1 rev1=1 rx=0x7f63f00426c0 tx=0x7f63f005e9b0).stop 2021-03-09T09:47:07.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:07.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0018770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 unknown :-1 s=CLOSED pgs=9663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:47:07.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:07.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:07.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:07.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:07.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:07.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:07.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:07.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:47:07.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:07.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001eb00 con 0x7f63d8008840 2021-03-09T09:47:07.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:07.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f6400146050 2021-03-09T09:47:07.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:07.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:07.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:07.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:07.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:07.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:07.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:47:07.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:07.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.409+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:07.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.410+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:07.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.410+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:07.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.410+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:07.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:07.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:07.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:07.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:07.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:07.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:47:07.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:47:07.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:07.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:07.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:47:07.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:07.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:07.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065bc0 con 0x7f6400146050 2021-03-09T09:47:07.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:47:07.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:07.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9677 cs=0 l=1 rev1=1 rx=0x7f63f402e010 tx=0x7f63f4034ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:07.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:47:07.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.411+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:07.426 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:07.413+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26515 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:07.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f6400146050 2021-03-09T09:47:07.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.413+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:07.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.413+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:07.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.413+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:07.427 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:07.427 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:07.428 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:07.428 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:07.428 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:07.428 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:07.429 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:07.429 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:07.429 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:07.429 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:07.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.414+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f6400146050 2021-03-09T09:47:07.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.414+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:07.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.414+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:47:07.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.415+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4053b70 con 0x7f6400146050 2021-03-09T09:47:07.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:07.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:07.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:08.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:08.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:08.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:08.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:08.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:08.537+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:09.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:09.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:09.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:10.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:10.021+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:10.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:10.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:10.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:10.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:11.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:11.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:11.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:11.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:47:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:12.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9677 cs=0 l=1 rev1=1 rx=0x7f63f402e010 tx=0x7f63f4034ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9677 cs=0 l=1 rev1=1 rx=0x7f63f402e010 tx=0x7f63f4034ec0).stop 2021-03-09T09:47:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:12.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:47:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:12.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:12.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aee0 con 0x7f63d8003400 2021-03-09T09:47:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c540 con 0x7f63d8008840 2021-03-09T09:47:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:12.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023b10 con 0x7f63d8005e00 2021-03-09T09:47:12.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:12.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:12.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.413+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:12.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.414+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:12.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.414+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:12.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.414+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:12.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.414+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:47:12.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.414+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:12.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.414+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:12.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.414+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:12.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.414+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:12.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.414+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:12.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:12.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:12.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:12.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:12.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:12.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:47:12.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:47:12.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:12.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:12.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027af0 con 0x7f63d8005e00 2021-03-09T09:47:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:47:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:47:12.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.415+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:12.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.416+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 secure :-1 s=READY pgs=9695 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f400e980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:12.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.416+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:47:12.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.416+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:12.429 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:12.417+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26552 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:12.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.418+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4061e90 con 0x7f63d8005e00 2021-03-09T09:47:12.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.418+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:12.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.418+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:12.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.418+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:12.430 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:12.430 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:12.431 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:12.431 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:12.431 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:12.431 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:12.432 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:12.432 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:12.432 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:12.432 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:12.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.420+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4061e90 con 0x7f63d8005e00 2021-03-09T09:47:12.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.420+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:12.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.420+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:47:12.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.421+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400eb30 con 0x7f63d8005e00 2021-03-09T09:47:12.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:12.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:12.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:13.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:13.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:13.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:13.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:14.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:14.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:14.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:14.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:15.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:15.022+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:15.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:15.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:15.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:15.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:16.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:16.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:16.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:16.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:17.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0018770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:17.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0018770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:17.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 secure :-1 s=READY pgs=9695 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f400e980).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:17.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 secure :-1 s=READY pgs=9695 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f400e980).stop 2021-03-09T09:47:17.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:17.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:17.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0018770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:17.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018770 unknown :-1 s=CLOSED pgs=9695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:17.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:47:17.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:17.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:17.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:17.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:17.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:17.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:17.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:17.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d080 con 0x7f63d8003400 2021-03-09T09:47:17.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:17.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:47:17.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:17.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ffe0 con 0x7f6400146050 2021-03-09T09:47:17.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:17.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:17.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.417+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:17.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.418+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:17.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.418+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:17.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.418+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:17.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.418+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:47:17.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.418+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:17.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.418+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:17.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.419+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:17.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.419+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:17.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.419+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:17.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:17.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:17.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:17.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:17.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:17.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:47:17.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:47:17.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:17.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:17.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:47:17.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:17.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:17.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a490 con 0x7f6400146050 2021-03-09T09:47:17.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:47:17.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.420+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:17.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.421+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9708 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:17.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.421+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:47:17.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.421+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:17.434 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:17.421+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26592 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:17.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.421+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f6400146050 2021-03-09T09:47:17.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.421+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:17.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.421+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:17.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.421+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:17.436 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:17.436 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:17.436 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:17.436 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:17.437 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:17.437 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:17.437 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:17.437 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:17.438 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:17.438 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:17.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.422+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f6400146050 2021-03-09T09:47:17.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.422+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:17.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.422+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:47:17.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.424+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f6400146050 2021-03-09T09:47:17.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:17.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:17.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:18.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:18.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:18.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:18.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:18.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:18.539+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:19.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:19.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:19.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:19.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:20.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:20.023+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:20.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:20.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:20.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:20.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:21.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:21.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:21.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:21.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:22.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:22.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:22.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9708 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:22.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9708 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc003a50).stop 2021-03-09T09:47:22.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:22.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:22.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:22.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9708 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:22.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:47:22.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:22.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:22.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:22.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:22.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:22.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:22.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:22.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63d8003400 2021-03-09T09:47:22.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:22.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aee0 con 0x7f63d8008840 2021-03-09T09:47:22.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:22.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008850 con 0x7f63d8005e00 2021-03-09T09:47:22.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:22.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:22.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:22.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:22.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:22.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.422+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:22.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.423+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:47:22.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.423+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:22.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.423+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.423+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.423+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.423+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:22.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:22.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:22.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:47:22.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:47:22.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:22.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:22.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f63d8005e00 2021-03-09T09:47:22.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:22.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:22.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T09:47:22.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:47:22.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.424+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:22.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018df0 secure :-1 s=READY pgs=9722 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc04fe50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:22.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.425+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:47:22.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.425+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:22.440 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:22.425+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26629 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:22.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:47:22.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.426+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:22.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.426+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:22.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.426+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:22.442 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:22.442 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:22.442 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:22.442 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:22.443 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:22.443 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:22.443 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:22.443 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:22.443 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:22.444 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:22.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046ae0 con 0x7f63d8005e00 2021-03-09T09:47:22.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.426+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:22.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.426+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:47:22.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.429+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8005e00 2021-03-09T09:47:22.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:22.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:22.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:23.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:23.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:23.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:23.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:24.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:24.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:24.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:24.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:25.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:25.024+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:25.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:25.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:25.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:25.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:26.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:26.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:26.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:26.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:27.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0018df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:27.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0018df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:27.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018df0 secure :-1 s=READY pgs=9722 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc04fe50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:27.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018df0 secure :-1 s=READY pgs=9722 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc04fe50).stop 2021-03-09T09:47:27.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:27.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:27.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0018df0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:27.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0018df0 unknown :-1 s=CLOSED pgs=9722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:27.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:47:27.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:27.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:27.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:27.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:27.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:27.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:27.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:27.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023b10 con 0x7f63d8003400 2021-03-09T09:47:27.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:27.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:47:27.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:27.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:47:27.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.425+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:27.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:27.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:27.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:27.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:27.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:27.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:47:27.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:27.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:27.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:27.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:27.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.426+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:27.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:27.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:27.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:27.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:27.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:27.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:47:27.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:47:27.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:27.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:27.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:47:27.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:27.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:27.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027af0 con 0x7f6400146050 2021-03-09T09:47:27.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:47:27.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:27.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9733 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4061f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:27.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:47:27.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.427+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:27.444 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:27.429+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26669 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:27.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4046b50 con 0x7f6400146050 2021-03-09T09:47:27.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.430+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:27.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.430+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:27.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.430+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:27.446 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:27.446 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:27.446 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:27.446 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:27.447 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:27.447 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:27.447 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:27.447 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:27.447 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:27.448 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:27.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.432+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046b50 con 0x7f6400146050 2021-03-09T09:47:27.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.432+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:27.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.432+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:47:27.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4046b50 con 0x7f6400146050 2021-03-09T09:47:27.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:27.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:27.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:28.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:28.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:28.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:28.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:28.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:28.540+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:29.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:29.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:29.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:29.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:30.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:30.025+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:30.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:30.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:30.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:30.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:31.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:31.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:31.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:31.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:32.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.429+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:32.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.429+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:32.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.429+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9733 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4061f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:32.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.429+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9733 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4061f00).stop 2021-03-09T09:47:32.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.429+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:32.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.429+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:32.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.429+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:32.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:32.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:47:32.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:32.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:32.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:32.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:32.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:32.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:32.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:32.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001eb00 con 0x7f63d8003400 2021-03-09T09:47:32.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:32.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aee0 con 0x7f63d8008840 2021-03-09T09:47:32.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:32.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T09:47:32.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:32.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:32.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:32.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:32.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:32.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.430+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:32.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.431+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:47:32.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.431+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:32.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.431+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:32.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.431+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:32.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.431+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:32.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.431+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:32.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:32.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:32.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:32.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:32.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:32.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:47:32.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:47:32.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:32.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:32.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c660 con 0x7f63d8005e00 2021-03-09T09:47:32.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:32.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:32.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f63d8005e00 2021-03-09T09:47:32.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:47:32.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.432+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:32.446 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:32.433+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26704 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:32.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.433+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e910 secure :-1 s=READY pgs=9752 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc02bfb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:32.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.434+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:47:32.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.434+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:32.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02ba50 con 0x7f63d8005e00 2021-03-09T09:47:32.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.434+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:32.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.434+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:32.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.434+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:32.449 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:32.449 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:32.449 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:32.450 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:32.450 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:32.450 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:32.450 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:32.450 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:32.451 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:32.451 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:32.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.435+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02b820 con 0x7f63d8005e00 2021-03-09T09:47:32.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.435+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:32.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.435+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:47:32.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.437+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0214a0 con 0x7f63d8005e00 2021-03-09T09:47:32.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:32.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:32.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:33.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:33.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:33.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:33.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:34.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:34.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:34.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:34.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:35.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:35.026+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:35.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:35.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:35.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:35.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:36.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:36.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:36.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:36.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:37.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:37.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:37.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e910 secure :-1 s=READY pgs=9752 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc02bfb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:37.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e910 secure :-1 s=READY pgs=9752 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc02bfb0).stop 2021-03-09T09:47:37.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:37.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:37.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000e910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:37.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e910 unknown :-1 s=CLOSED pgs=9752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:37.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:47:37.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:37.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:37.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:37.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:37.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:37.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:37.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:37.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d390 con 0x7f63d8003400 2021-03-09T09:47:37.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:37.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c3c0 con 0x7f63d8008840 2021-03-09T09:47:37.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:37.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015f00 con 0x7f6400146050 2021-03-09T09:47:37.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:37.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:37.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.434+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:37.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.435+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:37.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.435+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:37.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.435+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:37.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.435+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:47:37.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.435+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:37.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.435+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:37.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.435+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:37.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.435+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:37.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.435+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:37.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:37.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:37.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:37.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:37.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:37.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:47:37.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:47:37.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:37.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:37.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022fd0 con 0x7f6400146050 2021-03-09T09:47:37.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:37.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:37.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000bc00 con 0x7f6400146050 2021-03-09T09:47:37.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:47:37.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:37.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9763 cs=0 l=1 rev1=1 rx=0x7f63e0039af0 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:37.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:47:37.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.436+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:37.450 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:37.437+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26745 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:37.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00541f0 con 0x7f6400146050 2021-03-09T09:47:37.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.438+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:37.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.438+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:37.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.438+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:37.452 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:37.452 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:37.452 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:37.452 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:37.453 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:37.453 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:37.453 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:37.453 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:37.453 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:37.454 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:37.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00157f0 con 0x7f6400146050 2021-03-09T09:47:37.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.440+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:37.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.440+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:47:37.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.441+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00036b0 con 0x7f6400146050 2021-03-09T09:47:37.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:37.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:37.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:38.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:38.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:38.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:38.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:38.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:38.542+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:39.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:39.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:39.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:39.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:40.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:40.027+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:40.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:40.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:40.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:40.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:41.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:41.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:41.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:41.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:42.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:47:42.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:42.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9763 cs=0 l=1 rev1=1 rx=0x7f63e0039af0 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:42.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9763 cs=0 l=1 rev1=1 rx=0x7f63e0039af0 tx=0x7f63f0006610).stop 2021-03-09T09:47:42.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:42.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:42.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:42.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:42.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:47:42.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:42.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:42.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:42.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:42.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:42.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:42.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:42.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:47:42.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:42.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:47:42.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:42.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ed80 con 0x7f63d8005e00 2021-03-09T09:47:42.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:42.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:42.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:42.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:42.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.438+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:42.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.439+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.439+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:47:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.439+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.439+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:42.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.439+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:42.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.439+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:42.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.439+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:42.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.440+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:42.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.440+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:42.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.440+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:42.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:42.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:42.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:47:42.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:47:42.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:42.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:42.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014060 con 0x7f63d8005e00 2021-03-09T09:47:42.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:42.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:42.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022fd0 con 0x7f63d8005e00 2021-03-09T09:47:42.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:47:42.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:42.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:47:42.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:42.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.441+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 secure :-1 s=READY pgs=9779 cs=0 l=1 rev1=1 rx=0x7f63e0018360 tx=0x7f63f005df90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:42.454 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:42.442+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26783 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:42.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00039f0 con 0x7f63d8005e00 2021-03-09T09:47:42.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.443+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:42.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.443+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:42.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.443+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:42.456 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:42.456 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:42.456 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:42.457 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:42.457 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:42.457 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:42.457 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:42.457 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:42.458 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:42.458 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:42.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0003b50 con 0x7f63d8005e00 2021-03-09T09:47:42.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.443+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:42.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.443+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:47:42.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.445+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00039f0 con 0x7f63d8005e00 2021-03-09T09:47:42.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:42.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:42.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:43.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:43.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:43.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:43.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:44.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:44.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:44.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:44.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:45.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:45.028+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:45.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:45.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:45.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:45.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:46.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:46.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:47.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00243c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:47.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00243c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:47.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 secure :-1 s=READY pgs=9779 cs=0 l=1 rev1=1 rx=0x7f63e0018360 tx=0x7f63f005df90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:47.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 secure :-1 s=READY pgs=9779 cs=0 l=1 rev1=1 rx=0x7f63e0018360 tx=0x7f63f005df90).stop 2021-03-09T09:47:47.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:47.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:47.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00243c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:47.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 unknown :-1 s=CLOSED pgs=9779 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:47.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:47:47.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:47.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:47.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:47.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:47.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:47.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:47.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:47.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015f00 con 0x7f63d8003400 2021-03-09T09:47:47.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:47.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8008840 2021-03-09T09:47:47.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:47.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:47:47.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.442+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:47.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.443+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:47.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.443+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:47.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.443+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:47.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.443+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:47.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.443+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:47.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.443+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:47:47.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.443+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:47.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.443+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:47.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:47.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:47.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:47.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:47.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:47.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:47.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:47.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:47.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:47:47.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.444+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:47:47.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:47.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:47.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001d90 con 0x7f6400146050 2021-03-09T09:47:47.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:47:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:47:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9791 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4053f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:47.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:47:47.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.445+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:47.457 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:47.446+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26821 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:47.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.447+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f406a2d0 con 0x7f6400146050 2021-03-09T09:47:47.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.447+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:47.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.447+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:47.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.447+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:47.459 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:47.459 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:47.460 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:47.460 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:47.460 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:47.460 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:47.461 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:47.461 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:47.461 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:47.461 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:47.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f406a2d0 con 0x7f6400146050 2021-03-09T09:47:47.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.452+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:47.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.452+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:47:47.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.453+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405bbf0 con 0x7f6400146050 2021-03-09T09:47:47.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:47.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:47.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:48.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:48.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:48.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:48.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:48.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:48.543+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:49.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:49.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:49.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:49.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:50.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:50.029+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:50.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:50.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:50.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:50.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:51.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:51.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:51.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:51.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:52.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9791 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4053f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9791 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4053f30).stop 2021-03-09T09:47:52.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:52.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:52.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:52.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:52.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:47:52.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:52.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:52.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:52.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:52.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bb40 con 0x7f63d8003400 2021-03-09T09:47:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:52.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065d70 con 0x7f63d8008840 2021-03-09T09:47:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8005e00 2021-03-09T09:47:52.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:52.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:52.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:52.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:52.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:47:52.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:52.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:52.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:52.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:52.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:52.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:52.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.447+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:52.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:52.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:52.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:52.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:52.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:52.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:47:52.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:47:52.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:52.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:52.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067ee0 con 0x7f63d8005e00 2021-03-09T09:47:52.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:52.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:52.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c660 con 0x7f63d8005e00 2021-03-09T09:47:52.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:47:52.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.448+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:52.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.449+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:47:52.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.449+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:52.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.449+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 secure :-1 s=READY pgs=9802 cs=0 l=1 rev1=1 rx=0x7f63fc003910 tx=0x7f63fc046ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:52.462 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:52.450+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26860 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:52.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0215d0 con 0x7f63d8005e00 2021-03-09T09:47:52.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.451+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:52.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.451+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:52.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.451+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:52.463 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:52.464 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:52.464 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:52.464 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:52.464 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:52.465 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:52.465 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:52.465 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:52.465 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:52.465 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:52.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc021250 con 0x7f63d8005e00 2021-03-09T09:47:52.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.451+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:52.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.451+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:47:52.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f63d8005e00 2021-03-09T09:47:52.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:52.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:52.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:53.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:53.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:53.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:53.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:54.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:54.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:54.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:54.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:55.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:55.030+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:47:55.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:55.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:55.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:55.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:56.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:56.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:56.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:56.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00243c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:47:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00243c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:47:57.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 secure :-1 s=READY pgs=9802 cs=0 l=1 rev1=1 rx=0x7f63fc003910 tx=0x7f63fc046ae0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:47:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 secure :-1 s=READY pgs=9802 cs=0 l=1 rev1=1 rx=0x7f63fc003910 tx=0x7f63fc046ae0).stop 2021-03-09T09:47:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:47:57.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00243c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:47:57.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 unknown :-1 s=CLOSED pgs=9802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:47:57.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:47:57.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:57.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:47:57.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:57.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:47:57.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:47:57.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:47:57.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:57.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a270 con 0x7f63d8003400 2021-03-09T09:47:57.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:57.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8008840 2021-03-09T09:47:57.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:47:57.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f6400146050 2021-03-09T09:47:57.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:47:57.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:47:57.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:47:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.451+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:47:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.452+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:47:57.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.452+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:47:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.452+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:47:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.452+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:47:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.452+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:47:57.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.452+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:47:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.452+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:47:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:47:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:57.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:47:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:47:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:47:57.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:47:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f6400146050 2021-03-09T09:47:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:47:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:47:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001d90 con 0x7f6400146050 2021-03-09T09:47:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:47:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.453+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:57.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.454+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9817 cs=0 l=1 rev1=1 rx=0x7f63f400b0d0 tx=0x7f63f4017f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:47:57.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.454+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:47:57.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.454+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:47:57.467 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:47:57.454+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26898 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:47:57.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f6400146050 2021-03-09T09:47:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.455+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:47:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.455+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:47:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.455+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:47:57.469 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:47:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:47:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:47:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:47:57.470 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:47:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:47:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:47:57.471 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:47:57.471 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:47:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072e20 con 0x7f6400146050 2021-03-09T09:47:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.459+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:47:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.459+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:47:57.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f6400146050 2021-03-09T09:47:57.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:57.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:57.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:58.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:58.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:58.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:58.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:47:58.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:58.544+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:47:59.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:59.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:47:59.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:47:59.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:00.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:00.031+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:00.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:00.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:00.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:00.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:01.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:01.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:01.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:01.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9817 cs=0 l=1 rev1=1 rx=0x7f63f400b0d0 tx=0x7f63f4017f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9817 cs=0 l=1 rev1=1 rx=0x7f63f400b0d0 tx=0x7f63f4017f40).stop 2021-03-09T09:48:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9817 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:48:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:02.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:02.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:02.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:02.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a070 con 0x7f63d8003400 2021-03-09T09:48:02.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:02.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:48:02.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:02.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00304d0 con 0x7f63d8005e00 2021-03-09T09:48:02.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:02.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:02.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.455+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:02.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.456+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:02.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.456+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:02.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.456+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:02.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.456+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:48:02.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.456+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:02.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.456+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:02.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.456+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:02.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.456+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:02.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.456+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:02.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:02.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:02.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:02.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:02.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:02.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:48:02.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:48:02.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:02.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:02.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d30 con 0x7f63d8005e00 2021-03-09T09:48:02.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:02.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:02.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014060 con 0x7f63d8005e00 2021-03-09T09:48:02.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:48:02.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.457+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:02.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.458+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0001030 secure :-1 s=READY pgs=9831 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:02.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.458+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:48:02.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.458+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:02.471 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:02.459+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26937 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:02.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T09:48:02.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.460+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:02.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.460+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:02.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.460+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:02.473 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:02.473 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:02.473 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:02.474 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:02.474 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:02.474 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:02.474 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:02.475 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:02.475 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:02.475 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:02.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.463+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T09:48:02.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.463+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:02.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.463+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:48:02.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T09:48:02.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:02.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:02.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:03.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:03.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:03.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:03.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:04.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:04.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:04.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:04.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:05.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:05.032+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:05.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:05.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:05.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:05.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:06.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:06.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:06.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:06.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:07.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:07.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:07.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0001030 secure :-1 s=READY pgs=9831 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:07.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0001030 secure :-1 s=READY pgs=9831 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c30).stop 2021-03-09T09:48:07.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:07.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:07.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:07.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0001030 unknown :-1 s=CLOSED pgs=9831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:07.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:48:07.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:07.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:07.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:07.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:07.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:07.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:07.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:07.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b790 con 0x7f63d8003400 2021-03-09T09:48:07.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:07.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:48:07.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:07.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f6400146050 2021-03-09T09:48:07.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:07.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:07.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:07.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:07.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:07.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:07.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:48:07.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:07.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:07.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:07.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:07.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.460+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:07.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:07.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:07.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:07.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:07.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:07.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:48:07.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:48:07.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:07.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:07.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:48:07.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:07.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:07.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d30 con 0x7f6400146050 2021-03-09T09:48:07.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:48:07.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:07.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9842 cs=0 l=1 rev1=1 rx=0x7f63f0089070 tx=0x7f63f00066f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:07.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:48:07.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.461+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:07.475 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:07.463+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 26972 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:07.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.463+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f6400146050 2021-03-09T09:48:07.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.463+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:07.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.463+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:07.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.463+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:07.477 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:07.477 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:07.477 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:07.478 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:07.478 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:07.478 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:07.478 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:07.478 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:07.479 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:07.479 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:07.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.463+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:48:07.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.463+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:07.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.463+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:48:07.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.465+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e4a0 con 0x7f6400146050 2021-03-09T09:48:07.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:07.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:07.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:08.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:08.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:08.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:08.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:08.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:08.545+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:09.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:09.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:09.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:09.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:10.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:10.033+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:10.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:10.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:10.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:10.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:11.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:11.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:11.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:11.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:12.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:12.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:12.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9842 cs=0 l=1 rev1=1 rx=0x7f63f0089070 tx=0x7f63f00066f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:12.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9842 cs=0 l=1 rev1=1 rx=0x7f63f0089070 tx=0x7f63f00066f0).stop 2021-03-09T09:48:12.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:12.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:12.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:12.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:12.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:48:12.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:12.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:12.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:12.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:12.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:12.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:12.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:12.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00196d0 con 0x7f63d8003400 2021-03-09T09:48:12.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:12.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026750 con 0x7f63d8008840 2021-03-09T09:48:12.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:12.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001eb00 con 0x7f63d8005e00 2021-03-09T09:48:12.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:12.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:12.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:12.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:12.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:12.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:12.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:48:12.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:12.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:12.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:12.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:12.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.464+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:12.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:12.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:12.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:12.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:12.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:12.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:48:12.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:48:12.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:12.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:12.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8005e00 2021-03-09T09:48:12.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:12.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:12.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T09:48:12.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:48:12.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:12.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 secure :-1 s=READY pgs=9857 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f0028fc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:12.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:48:12.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.465+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:12.480 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:12.468+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27013 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.469+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T09:48:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.469+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.469+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:12.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.469+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:12.483 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:12.483 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:12.483 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:12.483 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:12.483 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:12.484 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:12.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.469+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00294e0 con 0x7f63d8005e00 2021-03-09T09:48:12.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.469+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:12.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.469+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:48:12.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.470+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63d8005e00 2021-03-09T09:48:12.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:12.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:12.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:13.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:13.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:13.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:13.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:14.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:14.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:14.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:14.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:15.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:15.034+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:15.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:15.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:15.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:15.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:16.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:16.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:16.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:16.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:17.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00243c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:48:17.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00243c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:17.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 secure :-1 s=READY pgs=9857 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f0028fc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 secure :-1 s=READY pgs=9857 cs=0 l=1 rev1=1 rx=0x7f63f0050be0 tx=0x7f63f0028fc0).stop 2021-03-09T09:48:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00243c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00243c0 unknown :-1 s=CLOSED pgs=9857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:48:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:17.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:17.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:17.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:17.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012590 con 0x7f63d8003400 2021-03-09T09:48:17.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T09:48:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00304d0 con 0x7f6400146050 2021-03-09T09:48:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.468+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:17.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:17.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:17.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:17.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:17.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:48:17.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:17.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:17.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:17.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:17.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.469+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:17.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:17.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:17.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:17.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:17.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:48:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:48:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001d90 con 0x7f6400146050 2021-03-09T09:48:17.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:17.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:17.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067ee0 con 0x7f6400146050 2021-03-09T09:48:17.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:48:17.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:17.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:48:17.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9875 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:17.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.471+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:17.482 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:17.472+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27051 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:17.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T09:48:17.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.473+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:17.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.473+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:17.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.473+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:17.484 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:17.484 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:17.485 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:17.485 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:17.485 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:17.485 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:17.485 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:17.486 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:17.486 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:17.486 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:17.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T09:48:17.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.476+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:17.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.476+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:48:17.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.477+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f6400146050 2021-03-09T09:48:17.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:17.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:17.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:18.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:18.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:18.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:18.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:18.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:18.546+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:19.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:19.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:19.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:19.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:20.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:20.035+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:20.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:20.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:20.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:20.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:21.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:21.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:21.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:21.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:22.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9875 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:22.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9875 cs=0 l=1 rev1=1 rx=0x7f63fc013eb0 tx=0x7f63fc04a740).stop 2021-03-09T09:48:22.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:22.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:22.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:22.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:22.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:48:22.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:22.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:22.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:22.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:48:22.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:22.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c350 con 0x7f63d8008840 2021-03-09T09:48:22.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:22.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8005e00 2021-03-09T09:48:22.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:22.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:22.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:22.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:22.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.473+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:22.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.474+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:22.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.474+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:48:22.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.474+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:22.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.474+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:22.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:22.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:22.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:22.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:22.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:22.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:22.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:22.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:22.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:48:22.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:48:22.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:22.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:22.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006baf0 con 0x7f63d8005e00 2021-03-09T09:48:22.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.475+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f63d8005e00 2021-03-09T09:48:22.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.476+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:48:22.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.476+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:22.515 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:22.476+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27090 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:22.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.477+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:48:22.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.477+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002da40 secure :-1 s=READY pgs=9892 cs=0 l=1 rev1=1 rx=0x7f63f4052240 tx=0x7f63f401acb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:22.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.477+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:22.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.478+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8005e00 2021-03-09T09:48:22.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.478+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:22.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.478+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:22.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.478+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:22.524 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:22.525 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:22.525 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:22.525 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:22.526 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:22.526 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:22.526 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:22.528 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:22.529 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:22.530 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:22.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f63d8005e00 2021-03-09T09:48:22.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.482+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:22.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.482+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:48:22.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8005e00 2021-03-09T09:48:22.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:22.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:22.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:23.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:23.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:23.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:23.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:24.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:24.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:24.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:24.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:25.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:25.036+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:25.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:25.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:25.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:25.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:26.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:26.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:26.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:26.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:27.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002da40 secure :-1 s=READY pgs=9892 cs=0 l=1 rev1=1 rx=0x7f63f4052240 tx=0x7f63f401acb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002da40 secure :-1 s=READY pgs=9892 cs=0 l=1 rev1=1 rx=0x7f63f4052240 tx=0x7f63f401acb0).stop 2021-03-09T09:48:27.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:27.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:27.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:27.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002da40 unknown :-1 s=CLOSED pgs=9892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:27.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:48:27.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:27.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:27.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cd10 con 0x7f63d8003400 2021-03-09T09:48:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065250 con 0x7f63d8008840 2021-03-09T09:48:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00196d0 con 0x7f6400146050 2021-03-09T09:48:27.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:27.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:27.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.477+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:27.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.478+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:27.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.478+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.478+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.478+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:48:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.478+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.478+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.478+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.478+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.478+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:27.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:27.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:27.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:27.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:48:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:48:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065d20 con 0x7f6400146050 2021-03-09T09:48:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001d90 con 0x7f6400146050 2021-03-09T09:48:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:48:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.479+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.480+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9899 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc03ff90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.480+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:48:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.480+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:27.495 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:27.481+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27128 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:27.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.481+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058410 con 0x7f6400146050 2021-03-09T09:48:27.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.481+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:27.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.481+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:27.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.481+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:27.497 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:27.497 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:27.497 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:27.497 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:27.497 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:27.498 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:27.499 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:27.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.481+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0581e0 con 0x7f6400146050 2021-03-09T09:48:27.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.481+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:27.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.481+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:48:27.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.483+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc021760 con 0x7f6400146050 2021-03-09T09:48:27.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:27.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:27.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:28.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:28.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:28.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:28.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:28.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:28.548+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.346+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000912) for more than 900000000 us, fault. 2021-03-09T09:48:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.346+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006b30).stop 2021-03-09T09:48:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.346+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.346+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e005e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.346+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e005e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.346+0000 7f6405d52700 10 monclient: get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T09:48:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.346+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e005e020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0006b30).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.346+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T09:48:29.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:29.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:29.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:30.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:30.037+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:30.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:30.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:30.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:30.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:31.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:31.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:31.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:31.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:32.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:48:32.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:32.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9899 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc03ff90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:32.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9899 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc03ff90).stop 2021-03-09T09:48:32.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:32.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:32.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:32.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:32.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:48:32.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:32.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:32.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:32.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:32.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00180d0 con 0x7f63d8003400 2021-03-09T09:48:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8008840 2021-03-09T09:48:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:48:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:32.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:32.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:32.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.482+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.483+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:32.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.483+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:48:32.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.483+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:32.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.483+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:32.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.484+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:32.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.484+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:32.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.484+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:32.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:32.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:48:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:48:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014060 con 0x7f63e0018390 2021-03-09T09:48:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:48:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:48:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:32.498 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27167 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=9923 cs=0 l=1 rev1=1 rx=0x7f63e000e220 tx=0x7f63f0031360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:48:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.485+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.486+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:48:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.486+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.486+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.486+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:32.501 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:32.501 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:32.501 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:32.501 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:32.501 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:32.502 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.489+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:48:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.489+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.489+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:48:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0039330 con 0x7f63e0018390 2021-03-09T09:48:32.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:32.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:32.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:33.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:33.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:33.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:33.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:34.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:34.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:34.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:35.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:35.038+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:35.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:35.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:35.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:35.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:36.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:36.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:36.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:36.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:37.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:37.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:37.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=9923 cs=0 l=1 rev1=1 rx=0x7f63e000e220 tx=0x7f63f0031360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:37.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 secure :-1 s=READY pgs=9923 cs=0 l=1 rev1=1 rx=0x7f63e000e220 tx=0x7f63f0031360).stop 2021-03-09T09:48:37.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:37.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:37.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00219b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:37.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00219b0 unknown :-1 s=CLOSED pgs=9923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:37.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:48:37.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:37.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63d8003400 2021-03-09T09:48:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00799d0 con 0x7f63d8008840 2021-03-09T09:48:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:37.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f6400146050 2021-03-09T09:48:37.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:37.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:37.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:37.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:37.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.486+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:37.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:37.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.488+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:48:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.488+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.488+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.488+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.488+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.488+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:37.506 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:37.489+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27205 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:37.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.489+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:37.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:48:37.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:48:37.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:37.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:37.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d230 con 0x7f6400146050 2021-03-09T09:48:37.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:37.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:37.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014060 con 0x7f6400146050 2021-03-09T09:48:37.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:48:37.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9935 cs=0 l=1 rev1=1 rx=0x7f63f001c860 tx=0x7f63f001c710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:48:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.490+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:48:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.491+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.491+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.491+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:37.514 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:37.514 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:37.514 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:37.515 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:37.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:48:37.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.494+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:37.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.494+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:48:37.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:48:37.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:37.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:37.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:38.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:38.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:38.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:38.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:38.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:38.550+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:39.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:39.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:39.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:39.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:40.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:40.039+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:40.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:40.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:40.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:40.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:41.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:41.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:41.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:41.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9935 cs=0 l=1 rev1=1 rx=0x7f63f001c860 tx=0x7f63f001c710).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:42.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9935 cs=0 l=1 rev1=1 rx=0x7f63f001c860 tx=0x7f63f001c710).stop 2021-03-09T09:48:42.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:42.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:42.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:42.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:42.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:48:42.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:42.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:42.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:42.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:42.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:42.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:42.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8003400 2021-03-09T09:48:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ed00 con 0x7f63d8008840 2021-03-09T09:48:42.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:42.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00196d0 con 0x7f63e0018390 2021-03-09T09:48:42.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.490+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:42.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.491+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:42.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.491+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:42.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.491+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:42.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.491+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:42.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.491+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:42.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.491+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:48:42.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.491+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.491+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:42.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:48:42.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:48:42.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:42.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:42.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001d90 con 0x7f63e0018390 2021-03-09T09:48:42.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:42.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:42.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d230 con 0x7f63e0018390 2021-03-09T09:48:42.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:48:42.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.492+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:42.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.493+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:48:42.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.493+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=9942 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f005dcb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:42.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.493+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:42.506 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:42.494+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27244 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:42.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:48:42.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.495+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:42.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.495+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:42.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.495+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:42.507 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:42.509 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63e0018390 2021-03-09T09:48:42.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.495+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:42.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.495+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:48:42.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e0018390 2021-03-09T09:48:42.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:42.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:42.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:43.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:43.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:43.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:43.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:44.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:44.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:44.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:44.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:45.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:45.040+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:45.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:45.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:45.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:45.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:46.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:46.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:46.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:46.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:47.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:47.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:47.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=9942 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f005dcb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:47.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=9942 cs=0 l=1 rev1=1 rx=0x7f63f0009600 tx=0x7f63f005dcb0).stop 2021-03-09T09:48:47.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:47.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:47.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:47.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=CLOSED pgs=9942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:47.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:48:47.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:47.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:47.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:47.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:47.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:47.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:47.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:47.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cd10 con 0x7f63d8003400 2021-03-09T09:48:47.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:47.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8008840 2021-03-09T09:48:47.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:47.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001eb00 con 0x7f6400146050 2021-03-09T09:48:47.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:47.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:47.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:47.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:47.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:47.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:47.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:48:47.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:47.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.495+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:47.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:47.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:47.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:47.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:47.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:47.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:47.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:47.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:47.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:48:47.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:48:47.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:47.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:47.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068560 con 0x7f6400146050 2021-03-09T09:48:47.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:47.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:47.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065d20 con 0x7f6400146050 2021-03-09T09:48:47.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:48:47.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.496+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:47.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.497+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:48:47.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.497+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:47.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.497+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9956 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc003880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:47.510 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:47.498+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27282 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:47.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0581e0 con 0x7f6400146050 2021-03-09T09:48:47.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.500+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:47.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.500+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:47.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.500+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:47.512 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:47.512 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:47.512 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:47.512 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:47.513 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:47.513 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:47.513 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:47.513 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:47.513 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:47.514 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:47.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.501+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058000 con 0x7f6400146050 2021-03-09T09:48:47.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.501+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:47.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.501+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:48:47.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T09:48:47.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:47.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:47.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:48.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:48.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:48.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:48.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:48.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:48.551+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:49.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:49.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:49.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:49.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:50.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:50.041+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:50.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:50.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:50.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:50.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:51.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:51.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:51.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:52.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:52.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:52.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9956 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc003880).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:52.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9956 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc003880).stop 2021-03-09T09:48:52.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:52.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:52.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:52.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:52.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:48:52.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0011d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:52.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:52.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:52.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:52.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:52.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:52.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:52.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T09:48:52.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:52.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065f70 con 0x7f63d8008840 2021-03-09T09:48:52.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:52.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068b50 con 0x7f63e0018390 2021-03-09T09:48:52.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:52.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027af0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:52.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:52.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:52.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:52.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0011d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:52.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:48:52.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:52.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.499+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:52.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:52.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:52.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:52.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:52.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:52.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:52.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:52.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:52.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:48:52.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:48:52.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:52.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:52.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T09:48:52.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:52.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:52.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006baf0 con 0x7f63e0018390 2021-03-09T09:48:52.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:48:52.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.500+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:52.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.501+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:48:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.501+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0011d80 secure :-1 s=READY pgs=9968 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f40632c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:52.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.501+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:52.514 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:52.503+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27321 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:52.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.503+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034e40 con 0x7f63e0018390 2021-03-09T09:48:52.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.503+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:52.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.503+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:52.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.503+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:52.516 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:52.516 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:52.516 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:52.517 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:52.517 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:52.517 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:52.518 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:52.518 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:52.518 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:52.518 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:52.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.505+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034e40 con 0x7f63e0018390 2021-03-09T09:48:52.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.505+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:52.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.505+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:48:52.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.505+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T09:48:52.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:52.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:52.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:53.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:53.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:53.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:53.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:54.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:54.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:54.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:54.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:55.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:55.042+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:48:55.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:55.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:55.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:55.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:56.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:56.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:56.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:56.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:57.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.503+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0011d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:48:57.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.503+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0011d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:48:57.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.503+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0011d80 secure :-1 s=READY pgs=9968 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f40632c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:48:57.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.503+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0011d80 secure :-1 s=READY pgs=9968 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f40632c0).stop 2021-03-09T09:48:57.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:57.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:48:57.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0011d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:48:57.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0011d80 unknown :-1 s=CLOSED pgs=9968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:48:57.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:48:57.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:57.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:48:57.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:57.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:48:57.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:48:57.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:48:57.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:57.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ed00 con 0x7f63d8003400 2021-03-09T09:48:57.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:57.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c8b0 con 0x7f63d8008840 2021-03-09T09:48:57.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:48:57.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:48:57.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:48:57.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:48:57.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:57.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:48:57.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.504+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:48:57.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.506+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:48:57.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.506+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:48:57.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.506+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:48:57.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.506+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:48:57.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.507+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:48:57.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.507+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:48:57.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.507+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:48:57.515 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27359 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:48:57.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:48:57.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:57.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:57.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:48:57.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:48:57.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:48:57.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:48:57.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:57.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:57.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.508+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:48:57.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.509+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:48:57.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.509+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:48:57.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.509+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068560 con 0x7f6400146050 2021-03-09T09:48:57.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.509+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:48:57.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.509+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:57.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.509+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9992 cs=0 l=1 rev1=1 rx=0x7f63fc047e30 tx=0x7f63fc012eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:48:57.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.509+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:48:57.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.509+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:48:57.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc046b50 con 0x7f6400146050 2021-03-09T09:48:57.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.511+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:48:57.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.511+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:48:57.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.511+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:48:57.522 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:48:57.522 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:48:57.523 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:57.523 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:48:57.523 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:48:57.523 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:48:57.523 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:48:57.524 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:48:57.524 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:48:57.524 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:48:57.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.513+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc046b50 con 0x7f6400146050 2021-03-09T09:48:57.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.513+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:48:57.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.513+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:48:57.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012240 con 0x7f6400146050 2021-03-09T09:48:57.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:57.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:57.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:58.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:58.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:58.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:58.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:48:58.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:58.552+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:48:59.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:59.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:48:59.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:48:59.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:00.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:00.043+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:00.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:00.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:00.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:00.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:01.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:01.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:01.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:01.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:02.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:02.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:02.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9992 cs=0 l=1 rev1=1 rx=0x7f63fc047e30 tx=0x7f63fc012eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:02.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=9992 cs=0 l=1 rev1=1 rx=0x7f63fc047e30 tx=0x7f63fc012eb0).stop 2021-03-09T09:49:02.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:02.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:02.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:02.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=9992 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:02.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:49:02.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:02.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:02.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:02.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:02.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:02.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:02.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:02.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8003400 2021-03-09T09:49:02.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:02.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027e00 con 0x7f63d8008840 2021-03-09T09:49:02.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:02.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010940 con 0x7f63e0018390 2021-03-09T09:49:02.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:02.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:02.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:02.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:02.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:02.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:02.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.509+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:49:02.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.510+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:02.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.510+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:02.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.510+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:02.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.510+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:02.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.510+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:02.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.510+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:02.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.510+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:02.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:02.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:02.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:02.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:49:02.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:49:02.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:02.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:02.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c8b0 con 0x7f63e0018390 2021-03-09T09:49:02.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:02.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:02.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:49:02.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:49:02.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:02.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=9997 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc052350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:02.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:49:02.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.511+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:02.524 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:02.512+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27398 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:02.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.513+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc008c90 con 0x7f63e0018390 2021-03-09T09:49:02.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.513+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:02.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.513+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:02.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.513+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:02.525 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:02.526 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:02.526 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:02.526 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:02.526 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:02.526 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:02.527 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:02.527 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:02.527 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:02.527 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:02.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.513+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058000 con 0x7f63e0018390 2021-03-09T09:49:02.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.513+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:02.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.513+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:49:02.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012240 con 0x7f63e0018390 2021-03-09T09:49:02.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:02.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:02.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:03.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:03.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:03.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:03.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:04.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:04.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:04.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:04.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:05.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:05.044+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:05.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:05.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:05.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:05.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:06.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:06.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:06.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:06.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:07.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.512+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:07.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.512+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:07.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.512+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=9997 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc052350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:07.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.512+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=9997 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc052350).stop 2021-03-09T09:49:07.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:07.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:07.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:07.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=CLOSED pgs=9997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:07.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:49:07.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:07.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:07.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00277b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:07.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T09:49:07.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:07.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025250 con 0x7f63d8008840 2021-03-09T09:49:07.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:07.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:49:07.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00277b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00277b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:49:07.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.513+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:07.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:07.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:07.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:07.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:07.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:07.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:07.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00277b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:07.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00277b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:07.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:49:07.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:49:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d6e0 con 0x7f6400146050 2021-03-09T09:49:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:07.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001d90 con 0x7f6400146050 2021-03-09T09:49:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:49:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.514+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.515+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10010 cs=0 l=1 rev1=1 rx=0x7f63f0041ff0 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:07.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.515+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:49:07.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.515+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:07.527 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:07.516+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27436 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.516+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005e870 con 0x7f6400146050 2021-03-09T09:49:07.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.516+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.517+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.517+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:07.529 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:07.530 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:07.530 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:07.530 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:07.530 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:07.530 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:07.531 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:07.531 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:07.531 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:07.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.517+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005e870 con 0x7f6400146050 2021-03-09T09:49:07.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.517+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:07.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.517+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:49:07.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001c340 con 0x7f6400146050 2021-03-09T09:49:07.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:07.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:07.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:08.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:08.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:08.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:08.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:08.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:08.553+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:09.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:09.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:10.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:10.045+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:10.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:10.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:10.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:10.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:11.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:11.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:11.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:11.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:12.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:12.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:12.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10010 cs=0 l=1 rev1=1 rx=0x7f63f0041ff0 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:12.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10010 cs=0 l=1 rev1=1 rx=0x7f63f0041ff0 tx=0x7f63f004d180).stop 2021-03-09T09:49:12.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:12.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:12.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:49:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:12.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:12.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:12.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00277b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:12.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:12.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:12.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ed00 con 0x7f63d8003400 2021-03-09T09:49:12.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:12.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068b50 con 0x7f63d8008840 2021-03-09T09:49:12.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:12.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000deb0 con 0x7f63e0018390 2021-03-09T09:49:12.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:12.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00277b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:12.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00277b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:12.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:12.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.517+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:12.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.519+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:12.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.519+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:49:12.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.519+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:12.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.519+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:12.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:12.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:12.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:12.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:12.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:12.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:12.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00277b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:12.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00277b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:12.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:49:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:49:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:49:12.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:12.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:12.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T09:49:12.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:49:12.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.520+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:12.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.521+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10032 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f4053f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:12.531 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:12.521+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27475 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:12.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.521+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:49:12.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.521+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:12.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.521+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401ad20 con 0x7f63e0018390 2021-03-09T09:49:12.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.521+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:12.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.521+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:12.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.521+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:12.533 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:12.534 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:12.534 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:12.534 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:12.534 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:12.535 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:12.535 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:12.535 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:12.535 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:12.535 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:12.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63e0018390 2021-03-09T09:49:12.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.522+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:12.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.522+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:49:12.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.523+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T09:49:12.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:12.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:12.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:13.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:13.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:13.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:13.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:14.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:14.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:14.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:14.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:15.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:15.046+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:15.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:15.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:15.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:15.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:16.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:16.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:16.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:16.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:17.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.521+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:17.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.521+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:17.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.521+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10032 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f4053f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:17.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.521+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10032 cs=0 l=1 rev1=1 rx=0x7f63f400f000 tx=0x7f63f4053f90).stop 2021-03-09T09:49:17.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:17.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:17.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:17.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=CLOSED pgs=10032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:17.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:49:17.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:17.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:17.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:17.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:17.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:17.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:17.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:17.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d960 con 0x7f63d8003400 2021-03-09T09:49:17.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:17.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:49:17.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:17.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e590 con 0x7f6400146050 2021-03-09T09:49:17.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:17.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:17.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:17.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:17.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:17.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:17.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:49:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.522+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:17.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.523+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:17.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.523+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:17.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.523+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:17.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:17.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:49:17.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:49:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015340 con 0x7f6400146050 2021-03-09T09:49:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:17.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:17.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:49:17.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:49:17.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:17.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10041 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4022520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:17.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:49:17.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.524+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:17.537 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:17.525+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27513 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:17.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.526+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034dd0 con 0x7f6400146050 2021-03-09T09:49:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.527+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.527+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.527+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:17.539 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:17.540 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:17.540 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:17.540 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:17.540 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:17.541 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:17.541 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:17.541 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:17.541 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:17.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:49:17.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.529+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:17.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.529+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:49:17.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.530+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022870 con 0x7f6400146050 2021-03-09T09:49:17.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:17.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:17.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:18.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:18.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:18.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:18.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:18.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:18.554+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:19.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:19.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:19.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:19.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:20.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:20.046+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:20.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:20.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:20.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:20.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:21.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:21.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:21.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:21.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:22.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:22.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:22.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10041 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4022520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:22.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10041 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4022520).stop 2021-03-09T09:49:22.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:22.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:22.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:22.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:22.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:49:22.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:22.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:22.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:22.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:22.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:22.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:22.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:22.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:49:22.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:22.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068b50 con 0x7f63d8008840 2021-03-09T09:49:22.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:22.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f680 con 0x7f63e0018390 2021-03-09T09:49:22.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.524+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:22.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.525+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:22.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.525+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:22.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.525+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:22.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.525+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:22.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.526+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:22.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.526+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:49:22.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.526+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:22.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.526+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:22.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:22.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:22.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:22.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:22.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:22.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:22.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:22.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:22.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:49:22.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:49:22.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:22.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:22.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001d90 con 0x7f63e0018390 2021-03-09T09:49:22.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:22.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:22.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c8b0 con 0x7f63e0018390 2021-03-09T09:49:22.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:49:22.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.527+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:22.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.528+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10059 cs=0 l=1 rev1=1 rx=0x7f63fc066fa0 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:22.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.528+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:49:22.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.528+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:22.547 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:22.528+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27552 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:22.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e0018390 2021-03-09T09:49:22.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.529+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:22.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.530+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:22.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.530+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:22.549 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:22.549 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:22.550 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:22.550 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:22.550 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:22.551 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:22.551 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:22.551 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:22.551 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:22.552 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:22.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.533+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e0018390 2021-03-09T09:49:22.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.533+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:22.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.533+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:49:22.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.535+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T09:49:22.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:22.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:22.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:23.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:23.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:23.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:23.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:24.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:24.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:24.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:24.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:25.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:25.047+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:25.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:25.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:25.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:25.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:26.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:26.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:26.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:26.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:27.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:27.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:27.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10059 cs=0 l=1 rev1=1 rx=0x7f63fc066fa0 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:27.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10059 cs=0 l=1 rev1=1 rx=0x7f63fc066fa0 tx=0x7f63fc003a50).stop 2021-03-09T09:49:27.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:27.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:27.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:27.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=CLOSED pgs=10059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:27.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:49:27.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:27.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:27.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:27.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:27.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:27.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:27.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:27.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d960 con 0x7f63d8003400 2021-03-09T09:49:27.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:27.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fad0 con 0x7f63d8008840 2021-03-09T09:49:27.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:27.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b360 con 0x7f6400146050 2021-03-09T09:49:27.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:27.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:27.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:27.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:27.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:49:27.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:27.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.529+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:27.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.530+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:27.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.530+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:27.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.530+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:27.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.530+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:27.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.530+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:27.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:27.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:27.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:27.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:27.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:27.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:49:27.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:49:27.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:27.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:27.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:49:27.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:27.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:27.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d6e0 con 0x7f6400146050 2021-03-09T09:49:27.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:49:27.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:27.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10067 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0030220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:27.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:49:27.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.531+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:27.547 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:27.533+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27590 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:27.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.534+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:49:27.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.534+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:27.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.534+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:27.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.534+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:27.548 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:27.549 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:27.549 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:27.549 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:27.549 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:27.549 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:27.550 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:27.550 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:27.550 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:27.551 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:27.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:49:27.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.539+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:27.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.539+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:49:27.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.540+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:49:27.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:27.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:27.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:28.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:28.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:28.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:28.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:28.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:28.555+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:29.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:29.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:29.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:29.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:30.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:30.048+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:30.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:30.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:30.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:30.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:31.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:31.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:31.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:31.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:32.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:32.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:32.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10067 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0030220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:32.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10067 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0030220).stop 2021-03-09T09:49:32.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:32.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:32.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:32.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:32.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:49:32.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:32.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:32.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:32.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:32.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:32.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:32.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:32.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ead0 con 0x7f63d8003400 2021-03-09T09:49:32.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:32.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ed00 con 0x7f63d8008840 2021-03-09T09:49:32.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:32.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:49:32.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:32.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.533+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:32.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.534+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:32.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.534+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:32.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.534+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:32.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.534+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:32.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.534+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:49:32.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.534+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:32.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.534+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:32.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:32.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:32.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:32.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:32.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:32.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:49:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:49:32.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:32.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:32.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002b5e0 con 0x7f63e0018390 2021-03-09T09:49:32.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:32.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:32.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015340 con 0x7f63e0018390 2021-03-09T09:49:32.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:49:32.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.535+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:32.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.536+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:49:32.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.536+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:32.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.536+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10081 cs=0 l=1 rev1=1 rx=0x7f63f4009c30 tx=0x7f63f4017f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:32.561 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:32.537+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27627 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:32.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405cee0 con 0x7f63e0018390 2021-03-09T09:49:32.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.538+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:32.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.538+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:32.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.538+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:32.565 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:32.565 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:32.566 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:32.566 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:32.567 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:32.567 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:32.567 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:32.568 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:32.568 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:32.569 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:32.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.544+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405cee0 con 0x7f63e0018390 2021-03-09T09:49:32.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.544+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:32.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.544+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:49:32.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.545+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0018390 2021-03-09T09:49:32.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:32.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:32.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:33.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:33.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:33.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:33.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:34.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:34.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:34.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:34.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:35.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:35.049+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:35.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:35.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:35.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:35.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:36.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:36.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:36.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:36.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:37.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:37.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:37.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10081 cs=0 l=1 rev1=1 rx=0x7f63f4009c30 tx=0x7f63f4017f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:37.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10081 cs=0 l=1 rev1=1 rx=0x7f63f4009c30 tx=0x7f63f4017f40).stop 2021-03-09T09:49:37.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:37.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:37.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:37.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=CLOSED pgs=10081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:37.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:49:37.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:37.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:37.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:37.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:37.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:37.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:37.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:37.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e160 con 0x7f63d8003400 2021-03-09T09:49:37.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:37.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fad0 con 0x7f63d8008840 2021-03-09T09:49:37.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:37.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068970 con 0x7f6400146050 2021-03-09T09:49:37.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:37.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:37.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.538+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:37.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.539+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:37.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.539+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:37.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.539+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:37.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.539+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:49:37.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.539+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:37.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.539+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:37.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.540+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:37.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.540+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:37.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.540+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:37.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:37.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:37.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:37.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:37.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:37.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:49:37.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:49:37.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:37.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:37.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00304b0 con 0x7f6400146050 2021-03-09T09:49:37.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:37.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:37.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001d90 con 0x7f6400146050 2021-03-09T09:49:37.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:49:37.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:37.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10099 cs=0 l=1 rev1=1 rx=0x7f63fc046f30 tx=0x7f63fc03ff80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:37.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:49:37.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.541+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:37.554 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:37.542+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27667 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:37.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.542+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:49:37.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.542+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:37.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.543+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:37.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.543+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:37.556 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:37.556 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:37.556 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:37.556 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:37.557 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:37.557 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:37.557 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:37.557 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:37.557 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:37.558 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:37.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.545+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:49:37.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.546+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:37.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.546+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:49:37.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:49:37.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:37.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:37.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:38.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:38.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:38.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:38.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:38.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:38.557+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:39.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:39.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:39.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:39.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:40.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:40.050+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:40.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:40.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:40.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:40.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:41.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:41.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:41.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:41.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:42.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.542+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:42.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.542+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:42.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.542+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10099 cs=0 l=1 rev1=1 rx=0x7f63fc046f30 tx=0x7f63fc03ff80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:42.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.542+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10099 cs=0 l=1 rev1=1 rx=0x7f63fc046f30 tx=0x7f63fc03ff80).stop 2021-03-09T09:49:42.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:42.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:42.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:42.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10099 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:42.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:49:42.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:42.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:42.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:42.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:42.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:42.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:42.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:42.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016bd0 con 0x7f63d8003400 2021-03-09T09:49:42.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:42.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b360 con 0x7f63d8008840 2021-03-09T09:49:42.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:42.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f3f0 con 0x7f63e0018390 2021-03-09T09:49:42.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:42.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.543+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:42.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.544+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:42.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.544+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:49:42.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.544+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:42.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.544+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:42.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.544+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:42.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.544+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:42.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.544+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:42.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:42.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:42.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:42.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:42.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:42.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:49:42.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:49:42.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:42.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:42.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c8b0 con 0x7f63e0018390 2021-03-09T09:49:42.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:42.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:42.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:49:42.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:49:42.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:42.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10110 cs=0 l=1 rev1=1 rx=0x7f63e0012990 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:42.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:49:42.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.545+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:42.558 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:42.546+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27706 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:42.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00287b0 con 0x7f63e0018390 2021-03-09T09:49:42.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.546+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:42.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.546+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:42.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.546+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:42.560 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:42.561 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:42.561 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:42.561 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:42.561 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:42.561 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:42.562 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:42.562 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:42.562 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:42.562 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:42.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007fe40 con 0x7f63e0018390 2021-03-09T09:49:42.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.547+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:42.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.547+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:49:42.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.549+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00287b0 con 0x7f63e0018390 2021-03-09T09:49:42.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:42.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:42.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:43.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:43.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:43.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:43.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:44.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:44.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:44.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:44.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:45.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:45.051+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:45.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:45.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:45.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:45.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:46.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:46.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:46.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:47.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.546+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:47.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.546+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:47.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.546+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10110 cs=0 l=1 rev1=1 rx=0x7f63e0012990 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:47.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10110 cs=0 l=1 rev1=1 rx=0x7f63e0012990 tx=0x7f63f004de00).stop 2021-03-09T09:49:47.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:47.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:47.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:47.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=CLOSED pgs=10110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:47.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:49:47.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:47.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:47.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:47.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:47.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:47.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:47.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:47.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ed00 con 0x7f63d8003400 2021-03-09T09:49:47.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:47.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fad0 con 0x7f63d8008840 2021-03-09T09:49:47.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:47.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400146050 2021-03-09T09:49:47.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:47.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:47.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:47.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:47.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:47.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:47.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:49:47.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:47.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.547+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:47.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.548+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:47.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.548+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:47.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.548+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:47.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:47.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:47.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:47.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:47.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:47.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:49:47.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:49:47.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:47.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:47.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:49:47.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:47.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:47.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c8b0 con 0x7f6400146050 2021-03-09T09:49:47.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:49:47.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.549+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:47.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.550+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:49:47.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.550+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10125 cs=0 l=1 rev1=1 rx=0x7f63e0022530 tx=0x7f63f007ef50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:47.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.550+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:47.581 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:47.551+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27744 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:47.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0088f00 con 0x7f6400146050 2021-03-09T09:49:47.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.551+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:47.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.551+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:47.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.551+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:47.582 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:47.583 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:47.583 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:47.583 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:47.583 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:47.583 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:47.584 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:47.584 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:47.584 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:47.584 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:47.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0088f00 con 0x7f6400146050 2021-03-09T09:49:47.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.554+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:47.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.554+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:49:47.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029c20 con 0x7f6400146050 2021-03-09T09:49:47.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:47.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:47.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:48.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:48.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:48.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:48.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:48.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:48.558+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:49.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:49.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:49.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:49.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:50.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:50.052+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:50.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:50.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:50.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:50.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:51.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:51.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:51.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:51.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:52.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:49:52.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:52.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10125 cs=0 l=1 rev1=1 rx=0x7f63e0022530 tx=0x7f63f007ef50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:52.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10125 cs=0 l=1 rev1=1 rx=0x7f63e0022530 tx=0x7f63f007ef50).stop 2021-03-09T09:49:52.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:52.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:52.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:52.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:49:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:52.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:52.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:52.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:52.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:52.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:52.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e700 con 0x7f63d8003400 2021-03-09T09:49:52.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:52.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00017b0 con 0x7f63d8008840 2021-03-09T09:49:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a9d0 con 0x7f63e0018390 2021-03-09T09:49:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:52.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.551+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:52.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.552+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:52.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.552+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:49:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.552+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.552+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.552+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.552+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.552+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.552+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.552+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:52.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:49:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:49:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:52.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012120 con 0x7f63e0018390 2021-03-09T09:49:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:52.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002b5e0 con 0x7f63e0018390 2021-03-09T09:49:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:49:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.553+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.554+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:49:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.554+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10136 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.554+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:52.569 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:52.555+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27773 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:52.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b2f0 con 0x7f63e0018390 2021-03-09T09:49:52.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.556+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:52.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.556+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:52.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.556+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:52.571 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:52.571 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:52.571 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:52.572 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:52.572 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:52.572 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:52.572 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:52.572 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:52.573 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:52.573 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:52.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b2f0 con 0x7f63e0018390 2021-03-09T09:49:52.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.556+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:52.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.556+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:49:52.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f63e0018390 2021-03-09T09:49:52.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:52.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:52.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:53.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:53.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:53.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:53.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:54.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:54.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:54.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:54.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:55.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:55.053+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:49:55.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:55.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:55.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:55.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:56.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:56.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:56.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:56.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:57.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:49:57.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:49:57.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10136 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003ad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:49:57.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 secure :-1 s=READY pgs=10136 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003ad0).stop 2021-03-09T09:49:57.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:57.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:49:57.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:49:57.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021070 unknown :-1 s=CLOSED pgs=10136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:49:57.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:49:57.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:57.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:49:57.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:57.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:49:57.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:49:57.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:49:57.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:57.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022620 con 0x7f63d8003400 2021-03-09T09:49:57.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:57.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:49:57.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:49:57.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ed00 con 0x7f6400146050 2021-03-09T09:49:57.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:49:57.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:49:57.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:49:57.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:57.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.555+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:49:57.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.556+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:49:57.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.556+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:49:57.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.556+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:49:57.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.556+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:49:57.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:49:57.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:49:57.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:49:57.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:49:57.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:57.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:57.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:49:57.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:49:57.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:49:57.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:49:57.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.557+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:57.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:57.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:49:57.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:49:57.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:49:57.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00304b0 con 0x7f6400146050 2021-03-09T09:49:57.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:49:57.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:57.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10152 cs=0 l=1 rev1=1 rx=0x7f63e000e4a0 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:49:57.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:49:57.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.558+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:49:57.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:57.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:57.572 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:49:57.559+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27821 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:49:57.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.560+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f6400146050 2021-03-09T09:49:57.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.560+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:49:57.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.560+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:49:57.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.560+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:49:57.574 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:49:57.574 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:49:57.574 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:57.574 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:49:57.575 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:49:57.575 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:49:57.575 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:49:57.575 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:49:57.576 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:49:57.576 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:49:57.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f6400146050 2021-03-09T09:49:57.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.562+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:49:57.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.562+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:49:57.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:57.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f6400146050 2021-03-09T09:49:58.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:58.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:58.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:58.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:49:58.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:58.559+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:49:59.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:59.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:49:59.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:49:59.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:00.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:00.054+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:00.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:00.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:00.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:00.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:01.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:01.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:01.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:01.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:02.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:02.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.559+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10152 cs=0 l=1 rev1=1 rx=0x7f63e000e4a0 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:02.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10152 cs=0 l=1 rev1=1 rx=0x7f63e000e4a0 tx=0x7f63fc003a50).stop 2021-03-09T09:50:02.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:50:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:02.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:02.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:02.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:02.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:02.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fca0 con 0x7f63d8003400 2021-03-09T09:50:02.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:02.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8008840 2021-03-09T09:50:02.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:02.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63e0018390 2021-03-09T09:50:02.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:02.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:02.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:02.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:02.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:50:02.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:02.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:02.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:02.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.560+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:02.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:02.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:02.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:02.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:02.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:02.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:02.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:02.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:02.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:50:02.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:50:02.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:02.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:02.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001d90 con 0x7f63e0018390 2021-03-09T09:50:02.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:02.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:02.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:50:02.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:50:02.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:02.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:50:02.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021050 secure :-1 s=READY pgs=10168 cs=0 l=1 rev1=1 rx=0x7f63f004d290 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:02.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.563+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:02.578 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:02.564+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27860 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:02.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.565+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:50:02.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.565+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:02.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.565+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:02.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.565+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:02.580 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:02.580 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:02.580 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:02.580 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:02.581 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:02.581 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:02.581 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:02.581 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:02.582 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:02.582 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:02.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:50:02.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.569+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:02.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036680 con 0x7f63e0018390 2021-03-09T09:50:02.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:02.569+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:50:03.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:03.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:03.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:03.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:04.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:04.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:04.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:04.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:05.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:05.055+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:05.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:05.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:05.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:05.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:06.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:06.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:06.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:06.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:07.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:07.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:07.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:07.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:07.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021050 secure :-1 s=READY pgs=10168 cs=0 l=1 rev1=1 rx=0x7f63f004d290 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:07.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021050 secure :-1 s=READY pgs=10168 cs=0 l=1 rev1=1 rx=0x7f63f004d290 tx=0x7f63f0013b80).stop 2021-03-09T09:50:07.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:07.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:07.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0021050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:07.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0021050 unknown :-1 s=CLOSED pgs=10168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:07.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:50:07.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:07.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:07.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:07.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:07.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:07.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:07.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:07.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T09:50:07.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:07.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63d8008840 2021-03-09T09:50:07.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:07.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a9d0 con 0x7f6400146050 2021-03-09T09:50:07.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:07.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00275b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:07.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:07.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:07.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:07.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:50:07.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:07.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:07.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.565+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:07.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.566+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:07.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.566+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:07.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.566+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:07.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:07.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:07.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:07.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00275b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:07.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:07.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:50:07.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:50:07.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:07.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:07.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c8b0 con 0x7f6400146050 2021-03-09T09:50:07.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:07.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:07.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:50:07.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:50:07.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.567+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:07.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.568+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:50:07.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.568+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:07.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.568+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10181 cs=0 l=1 rev1=1 rx=0x7f63e0023b40 tx=0x7f63fc0334d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:07.580 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:07.568+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27898 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:07.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc008c90 con 0x7f6400146050 2021-03-09T09:50:07.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.571+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:07.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.571+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:07.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.571+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:07.582 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:07.582 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:07.582 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:07.583 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:07.583 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:07.583 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:07.583 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:07.583 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:07.584 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:07.584 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:07.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc008c90 con 0x7f6400146050 2021-03-09T09:50:07.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.571+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:07.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.571+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:50:07.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:07.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:50:08.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:08.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:08.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:08.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:08.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:08.560+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:09.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:09.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:09.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:09.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:10.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:10.056+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:10.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:10.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:10.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:10.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:11.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:11.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:11.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:12.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10181 cs=0 l=1 rev1=1 rx=0x7f63e0023b40 tx=0x7f63fc0334d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:12.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10181 cs=0 l=1 rev1=1 rx=0x7f63e0023b40 tx=0x7f63fc0334d0).stop 2021-03-09T09:50:12.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:12.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:12.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:50:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:12.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:12.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00017b0 con 0x7f63d8003400 2021-03-09T09:50:12.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:12.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:50:12.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:12.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fca0 con 0x7f63e0018390 2021-03-09T09:50:12.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:12.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00275b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:12.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:12.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:12.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:50:12.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:12.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:12.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.569+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:12.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:12.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:12.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:12.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:12.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:12.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:12.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:12.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00275b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:12.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:12.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:50:12.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:50:12.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:12.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:12.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bf70 con 0x7f63e0018390 2021-03-09T09:50:12.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:12.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:12.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012120 con 0x7f63e0018390 2021-03-09T09:50:12.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:50:12.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.570+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:12.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.571+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10189 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404ce40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:12.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.571+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:50:12.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.571+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:12.584 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:12.573+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27937 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:12.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0018390 2021-03-09T09:50:12.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.573+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:12.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.573+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:12.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.573+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:12.586 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:12.586 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:12.587 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:12.587 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:12.587 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:12.587 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:12.587 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:12.588 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0018390 2021-03-09T09:50:12.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.573+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.573+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:50:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:12.574+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0018390 2021-03-09T09:50:13.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:13.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:13.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:13.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:14.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:14.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:14.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:14.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:15.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:15.057+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:15.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:15.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:15.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:15.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:16.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:16.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:16.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:16.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:17.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:17.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:17.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:17.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:17.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10189 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404ce40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:17.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10189 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f404ce40).stop 2021-03-09T09:50:17.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:17.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:17.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:17.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:17.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:50:17.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:17.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:17.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:17.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:17.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:17.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:17.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:17.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066170 con 0x7f63d8003400 2021-03-09T09:50:17.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:17.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001be30 con 0x7f63d8008840 2021-03-09T09:50:17.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:17.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f6400146050 2021-03-09T09:50:17.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:17.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00275b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:17.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.573+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:17.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:17.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:17.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:17.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:50:17.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:17.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:17.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:17.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:17.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:17.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:17.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:17.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:17.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00275b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:17.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:17.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:50:17.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:50:17.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:17.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:17.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f660 con 0x7f6400146050 2021-03-09T09:50:17.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:17.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:17.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c8b0 con 0x7f6400146050 2021-03-09T09:50:17.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:50:17.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.574+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:17.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.575+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10202 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc075f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:17.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.575+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:50:17.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.575+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:17.598 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:17.578+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 27975 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:17.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.578+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0214c0 con 0x7f6400146050 2021-03-09T09:50:17.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.578+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:17.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.578+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:17.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.578+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:17.601 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:17.602 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:17.602 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:17.602 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:17.603 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:17.603 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:17.603 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:17.604 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:17.604 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:17.604 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:17.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.578+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0214c0 con 0x7f6400146050 2021-03-09T09:50:17.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.578+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:17.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.578+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:50:17.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:17.580+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:50:18.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:18.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:18.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:18.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:18.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:18.561+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:19.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:19.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:19.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:19.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:20.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:20.058+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:20.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:20.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:20.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:20.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:21.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:21.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:21.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:21.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:22.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:22.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10202 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc075f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10202 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc075f70).stop 2021-03-09T09:50:22.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:22.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:22.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:22.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:22.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:50:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63d8003400 2021-03-09T09:50:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dc10 con 0x7f63d8008840 2021-03-09T09:50:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024b40 con 0x7f63e0018390 2021-03-09T09:50:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00275b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:22.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.578+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:22.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.580+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:22.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.580+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:50:22.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.580+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:22.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.580+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.580+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.580+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.580+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00275b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00275b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:50:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:50:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:50:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f660 con 0x7f63e0018390 2021-03-09T09:50:22.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:50:22.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.581+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:22.591 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:22.582+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28014 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:22.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.582+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:50:22.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.582+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:22.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.582+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10221 cs=0 l=1 rev1=1 rx=0x7f63fc046b20 tx=0x7f63fc0214c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:22.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.583+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02d950 con 0x7f63e0018390 2021-03-09T09:50:22.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.583+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:22.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.583+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:22.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.583+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:22.594 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:22.594 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:22.594 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:22.594 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:22.595 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:22.595 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:22.595 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:22.595 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:22.595 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:22.596 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02d950 con 0x7f63e0018390 2021-03-09T09:50:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.588+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.588+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:50:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:22.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63e0018390 2021-03-09T09:50:23.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:23.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:23.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:23.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:24.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:24.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:24.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:24.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:25.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:25.059+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:25.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:25.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:25.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:25.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:26.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:26.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:26.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:26.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:27.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:27.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:27.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:27.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:27.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10221 cs=0 l=1 rev1=1 rx=0x7f63fc046b20 tx=0x7f63fc0214c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10221 cs=0 l=1 rev1=1 rx=0x7f63fc046b20 tx=0x7f63fc0214c0).stop 2021-03-09T09:50:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:50:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:50:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024260 con 0x7f63d8008840 2021-03-09T09:50:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e0b0 con 0x7f6400146050 2021-03-09T09:50:27.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:27.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:27.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:27.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:27.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.583+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:27.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:27.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:50:27.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:27.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:27.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:27.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:27.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:27.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:27.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:27.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:27.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:27.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:27.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:50:27.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:50:27.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:27.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012120 con 0x7f6400146050 2021-03-09T09:50:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:50:27.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:50:27.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.584+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:27.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.585+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:50:27.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.585+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:27.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.585+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10232 cs=0 l=1 rev1=1 rx=0x7f63e005d3e0 tx=0x7f63fc01ef90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:27.599 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:27.586+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28052 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:27.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066d50 con 0x7f6400146050 2021-03-09T09:50:27.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.587+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:27.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.587+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:27.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.587+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:27.601 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:27.601 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:27.601 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:27.602 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:27.602 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:27.602 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:27.602 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:27.603 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:27.603 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:27.603 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:27.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc066d50 con 0x7f6400146050 2021-03-09T09:50:27.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.587+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:27.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.587+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:50:27.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:27.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T09:50:28.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:28.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:28.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:28.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:28.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:28.563+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:29.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:29.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:29.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:29.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:30.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:30.060+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:30.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:30.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:30.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:30.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:31.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:31.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:31.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:31.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:32.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:32.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:32.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:32.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:32.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10232 cs=0 l=1 rev1=1 rx=0x7f63e005d3e0 tx=0x7f63fc01ef90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:32.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10232 cs=0 l=1 rev1=1 rx=0x7f63e005d3e0 tx=0x7f63fc01ef90).stop 2021-03-09T09:50:32.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:32.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:32.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:32.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:50:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:32.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:32.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:32.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:32.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T09:50:32.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:32.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8008840 2021-03-09T09:50:32.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:32.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63e0018390 2021-03-09T09:50:32.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:32.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:32.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:32.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:32.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:50:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:32.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:32.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.587+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:32.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:32.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:50:32.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:50:32.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:32.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0009370 con 0x7f63e0018390 2021-03-09T09:50:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bf70 con 0x7f63e0018390 2021-03-09T09:50:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:50:32.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:32.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10245 cs=0 l=1 rev1=1 rx=0x7f63f4017f80 tx=0x7f63f4003c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:32.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:50:32.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.588+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:32.603 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:32.591+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28091 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:32.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.591+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63e0018390 2021-03-09T09:50:32.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.591+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:32.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.591+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:32.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.591+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:32.605 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:32.605 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:32.605 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:32.606 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:32.606 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:32.606 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:32.606 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:32.606 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:32.607 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:32.607 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:32.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.591+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005480 con 0x7f63e0018390 2021-03-09T09:50:32.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.591+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:32.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.591+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:50:32.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:32.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63e0018390 2021-03-09T09:50:33.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:33.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:33.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:33.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:34.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:34.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:34.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:35.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:35.061+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:35.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:35.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:35.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:35.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:36.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:36.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:36.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:36.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:37.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:37.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:37.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:37.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:37.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10245 cs=0 l=1 rev1=1 rx=0x7f63f4017f80 tx=0x7f63f4003c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:37.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10245 cs=0 l=1 rev1=1 rx=0x7f63f4017f80 tx=0x7f63f4003c20).stop 2021-03-09T09:50:37.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:37.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:37.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:37.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:37.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:50:37.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:37.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:37.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:37.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:37.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:37.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:37.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:37.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00237d0 con 0x7f63d8003400 2021-03-09T09:50:37.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:37.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014280 con 0x7f63d8008840 2021-03-09T09:50:37.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:37.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:50:37.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:37.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:37.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:37.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:37.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:37.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:37.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:50:37.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:37.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:37.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:37.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:37.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.592+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:37.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:37.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:37.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:50:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:50:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:50:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0009370 con 0x7f6400146050 2021-03-09T09:50:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:50:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.593+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.594+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:50:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.594+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:37.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.594+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10260 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4034a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:37.607 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:37.595+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28129 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:37.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034c30 con 0x7f6400146050 2021-03-09T09:50:37.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.596+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:37.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.596+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:37.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.596+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:37.609 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:37.609 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:37.611 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:37.611 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:37.611 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:37.611 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:37.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.597+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404be90 con 0x7f6400146050 2021-03-09T09:50:37.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.597+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:37.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.597+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:50:37.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:37.598+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40343c0 con 0x7f6400146050 2021-03-09T09:50:38.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:38.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:38.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:38.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:38.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:38.564+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:39.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:39.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:39.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:39.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:40.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:40.062+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:40.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:40.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:40.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:40.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:41.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:41.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:41.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:41.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:42.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:42.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:50:42.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10260 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4034a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10260 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4034a00).stop 2021-03-09T09:50:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:42.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:42.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:42.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:50:42.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:42.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:42.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:42.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:42.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015560 con 0x7f63d8003400 2021-03-09T09:50:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:42.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014060 con 0x7f63d8008840 2021-03-09T09:50:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63e0018390 2021-03-09T09:50:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:42.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:50:42.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.596+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.597+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.597+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.597+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:42.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.597+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:42.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:42.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:42.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:42.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:42.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:50:42.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:50:42.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:42.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f63e0018390 2021-03-09T09:50:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001d90 con 0x7f63e0018390 2021-03-09T09:50:42.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:50:42.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:42.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10276 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:42.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:50:42.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.598+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:42.612 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:42.600+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28170 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:42.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:50:42.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.601+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:42.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.601+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.601+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:42.615 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:42.615 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:42.615 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:42.616 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:42.616 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:42.616 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:42.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.603+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:50:42.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.603+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:42.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.603+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:50:42.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:42.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e290 con 0x7f63e0018390 2021-03-09T09:50:43.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:43.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:43.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:43.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:44.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:44.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:44.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:44.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:45.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:45.063+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:45.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:45.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:45.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:45.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:46.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:46.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:46.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:46.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:47.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:47.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:47.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:50:47.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:47.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10276 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:47.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10276 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000c30).stop 2021-03-09T09:50:47.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:47.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:47.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:47.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:47.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:50:47.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:47.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:47.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:47.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:47.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:47.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:47.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:47.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63d8003400 2021-03-09T09:50:47.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:47.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8008840 2021-03-09T09:50:47.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:47.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:50:47.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:47.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:47.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:47.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:47.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:47.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:47.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:50:47.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:47.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.601+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:47.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:47.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:47.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:47.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:47.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:47.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:47.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:47.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:47.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:50:47.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:50:47.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:47.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:47.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d790 con 0x7f6400146050 2021-03-09T09:50:47.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:47.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:47.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:50:47.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:50:47.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.602+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:47.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.603+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10286 cs=0 l=1 rev1=1 rx=0x7f63f403b010 tx=0x7f63f4003b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:47.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.603+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:50:47.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.603+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:47.617 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:47.605+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28206 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:47.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40035e0 con 0x7f6400146050 2021-03-09T09:50:47.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.605+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:47.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.605+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:47.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.605+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:47.619 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:47.619 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:47.619 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:47.620 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:47.620 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:47.620 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:47.620 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:47.621 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:47.621 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:47.621 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:47.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40035e0 con 0x7f6400146050 2021-03-09T09:50:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.608+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.608+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:50:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:47.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f6400146050 2021-03-09T09:50:48.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:48.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:48.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:48.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:48.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:48.566+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:49.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:49.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:49.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:49.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:50.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:50.064+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:50.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:50.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:50.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:50.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:51.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:51.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:51.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:51.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:52.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:52.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:52.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:52.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:52.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10286 cs=0 l=1 rev1=1 rx=0x7f63f403b010 tx=0x7f63f4003b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:52.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10286 cs=0 l=1 rev1=1 rx=0x7f63f403b010 tx=0x7f63f4003b40).stop 2021-03-09T09:50:52.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:52.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:52.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:52.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:52.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:50:52.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:52.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:52.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:52.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:52.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:52.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:52.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:52.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065a80 con 0x7f63d8003400 2021-03-09T09:50:52.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:52.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fad0 con 0x7f63d8008840 2021-03-09T09:50:52.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:52.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021720 con 0x7f63e0018390 2021-03-09T09:50:52.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:52.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:52.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:50:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:52.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:52.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:52.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:52.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.606+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:52.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:52.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:52.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:52.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:52.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:52.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:50:52.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:50:52.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:52.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:52.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002b8a0 con 0x7f63e0018390 2021-03-09T09:50:52.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:52.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:52.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012120 con 0x7f63e0018390 2021-03-09T09:50:52.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:50:52.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:52.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:50:52.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:52.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.607+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10301 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:52.621 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:52.609+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28246 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:52.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T09:50:52.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.610+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:52.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.610+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:52.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.610+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:52.623 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:52.623 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:52.623 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:52.623 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:52.623 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:52.624 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:52.624 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:52.624 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:52.624 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:52.625 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:52.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0018390 2021-03-09T09:50:52.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.610+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:52.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.610+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:50:52.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:52.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T09:50:53.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:53.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:53.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:53.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:54.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:54.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:54.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:54.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:55.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:55.065+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:50:55.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:55.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:55.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:55.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:56.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:56.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:56.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:56.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:57.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:57.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:57.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:50:57.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:50:57.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10301 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:50:57.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10301 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a6d0).stop 2021-03-09T09:50:57.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:57.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:50:57.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:50:57.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:50:57.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:50:57.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:57.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:50:57.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:57.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:50:57.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:50:57.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:50:57.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:57.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T09:50:57.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:57.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fea0 con 0x7f63d8008840 2021-03-09T09:50:57.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:50:57.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:50:57.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.610+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:50:57.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:50:57.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:50:57.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:50:57.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:50:57.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:50:57.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:50:57.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:50:57.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:57.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:50:57.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:50:57.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.611+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:50:57.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:50:57.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:57.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:57.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:50:57.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:50:57.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:50:57.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:50:57.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:57.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:57.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ccb0 con 0x7f6400146050 2021-03-09T09:50:57.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:50:57.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:50:57.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d790 con 0x7f6400146050 2021-03-09T09:50:57.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:50:57.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:57.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10316 cs=0 l=1 rev1=1 rx=0x7f63f406acf0 tx=0x7f63f40036c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:50:57.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:50:57.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.612+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:50:57.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.614+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f6400146050 2021-03-09T09:50:57.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.614+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:50:57.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.614+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:50:57.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.614+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:50:57.629 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:50:57.629 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:50:57.629 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:57.629 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:50:57.630 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:50:57.630 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:50:57.630 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:50:57.630 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:50:57.631 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:50:57.631 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:50:57.631 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:50:57.614+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28285 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:50:57.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:50:57.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.615+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:50:57.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.615+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:50:57.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:57.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f6400146050 2021-03-09T09:50:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:58.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:58.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:50:58.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:58.567+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:50:59.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:59.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:50:59.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:50:59.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:00.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:00.066+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:00.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:00.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:00.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:00.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:01.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:01.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:01.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:01.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:02.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:02.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:02.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:02.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:02.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10316 cs=0 l=1 rev1=1 rx=0x7f63f406acf0 tx=0x7f63f40036c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:02.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10316 cs=0 l=1 rev1=1 rx=0x7f63f406acf0 tx=0x7f63f40036c0).stop 2021-03-09T09:51:02.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:02.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:02.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:02.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:02.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:51:02.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:02.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:02.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e001a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:02.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:02.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:02.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:02.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:02.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:51:02.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:02.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021720 con 0x7f63d8008840 2021-03-09T09:51:02.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:02.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0018390 2021-03-09T09:51:02.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:02.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:02.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:02.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e001a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:02.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:51:02.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:02.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.615+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:02.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.616+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:02.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.616+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:02.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.616+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:02.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.616+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:02.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.616+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:02.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:02.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:02.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:02.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:02.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:02.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:51:02.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:51:02.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:02.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:02.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:51:02.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:02.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:02.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002b8a0 con 0x7f63e0018390 2021-03-09T09:51:02.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:51:02.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:02.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.617+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e001a4a0 secure :-1 s=READY pgs=10333 cs=0 l=1 rev1=1 rx=0x7f63e0015b00 tx=0x7f63fc008d30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:02.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.618+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:51:02.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.618+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:02.629 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:02.618+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28322 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:02.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T09:51:02.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.619+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:02.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.619+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:02.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.619+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:02.631 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:02.631 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:02.632 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:02.632 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:02.632 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:02.633 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:02.633 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:02.633 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:02.633 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:02.633 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:02.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.623+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc033580 con 0x7f63e0018390 2021-03-09T09:51:02.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.623+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:02.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.624+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:51:02.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:02.624+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T09:51:03.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:03.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:03.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:03.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:04.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:04.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:04.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:04.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:05.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:05.067+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:05.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:05.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:05.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:05.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:06.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:06.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:06.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:06.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:07.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:07.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:07.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e001a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:07.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e001a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:07.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e001a4a0 secure :-1 s=READY pgs=10333 cs=0 l=1 rev1=1 rx=0x7f63e0015b00 tx=0x7f63fc008d30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e001a4a0 secure :-1 s=READY pgs=10333 cs=0 l=1 rev1=1 rx=0x7f63e0015b00 tx=0x7f63fc008d30).stop 2021-03-09T09:51:07.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e001a4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:07.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e001a4a0 unknown :-1 s=CLOSED pgs=10333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:07.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:51:07.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:07.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:07.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:07.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:07.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:07.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:07.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T09:51:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:51:07.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:07.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400146050 2021-03-09T09:51:07.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.619+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:07.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:07.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:07.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:07.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:07.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:07.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:51:07.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:07.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:07.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:07.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:07.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.620+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:07.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:07.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:07.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:07.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:07.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:07.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:51:07.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:51:07.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026700 con 0x7f6400146050 2021-03-09T09:51:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:07.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ccb0 con 0x7f6400146050 2021-03-09T09:51:07.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:51:07.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.621+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:07.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.622+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10345 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4061890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:07.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.622+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:51:07.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.622+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:07.646 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:07.623+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28362 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:07.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.623+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40728d0 con 0x7f6400146050 2021-03-09T09:51:07.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.623+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.623+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.623+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:07.649 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:07.650 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:07.650 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:07.650 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:07.650 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:07.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.624+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40728d0 con 0x7f6400146050 2021-03-09T09:51:07.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.624+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:07.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.624+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:51:07.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:07.625+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f6400146050 2021-03-09T09:51:08.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:08.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:08.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:08.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:08.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:08.569+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:09.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:09.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:09.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:09.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:10.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:10.068+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:10.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:10.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:10.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:10.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:11.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:11.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:11.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:11.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:12.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:12.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:51:12.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:12.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10345 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4061890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:12.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10345 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4061890).stop 2021-03-09T09:51:12.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:12.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:12.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:12.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:12.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:51:12.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:12.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:12.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:12.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:12.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:51:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022180 con 0x7f63d8008840 2021-03-09T09:51:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002eeb0 con 0x7f63e0018390 2021-03-09T09:51:12.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:51:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.624+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.625+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.625+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:12.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.625+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.625+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.625+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.625+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:12.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:12.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:51:12.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:51:12.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:12.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:12.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012120 con 0x7f63e0018390 2021-03-09T09:51:12.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:12.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:12.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026700 con 0x7f63e0018390 2021-03-09T09:51:12.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:51:12.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:12.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10357 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f40462f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:12.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.626+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:51:12.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.627+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:12.640 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:12.628+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28399 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:12.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402d4b0 con 0x7f63e0018390 2021-03-09T09:51:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.628+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.628+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.628+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:12.642 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:12.643 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072630 con 0x7f63e0018390 2021-03-09T09:51:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.628+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.628+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:51:12.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:12.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f63e0018390 2021-03-09T09:51:13.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:13.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:13.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:13.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:14.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:14.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:14.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:14.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:15.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:15.069+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:15.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:15.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:15.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:16.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:16.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:16.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:16.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:17.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:17.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:17.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:17.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:17.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10357 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f40462f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:17.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10357 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f40462f0).stop 2021-03-09T09:51:17.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:17.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:17.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:17.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:17.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:51:17.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:17.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:17.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:17.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:17.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:17.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:17.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:17.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028630 con 0x7f63d8003400 2021-03-09T09:51:17.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:17.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068b50 con 0x7f63d8008840 2021-03-09T09:51:17.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:17.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d530 con 0x7f6400146050 2021-03-09T09:51:17.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:17.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.628+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:17.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:17.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:17.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:17.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:17.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:51:17.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:17.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:17.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:17.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:17.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:17.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:17.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:17.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:17.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:17.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:17.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:51:17.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:51:17.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:17.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:17.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013280 con 0x7f6400146050 2021-03-09T09:51:17.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:17.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:17.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f6400146050 2021-03-09T09:51:17.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:51:17.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:17.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10370 cs=0 l=1 rev1=1 rx=0x7f63f0058070 tx=0x7f63f004d110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:17.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:51:17.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.630+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:17.645 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:17.633+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28439 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:17.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.633+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:51:17.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.633+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:17.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.633+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:17.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.633+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:17.646 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:17.646 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:17.647 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:17.647 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:17.647 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:17.647 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:17.648 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:17.648 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:17.648 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:17.648 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:17.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.633+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00541f0 con 0x7f6400146050 2021-03-09T09:51:17.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.633+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:17.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.633+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:51:17.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:17.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:51:18.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:18.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:18.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:18.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:18.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:18.570+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:19.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:19.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:19.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:19.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:20.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:20.070+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:20.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:20.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:20.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:20.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:21.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:21.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:21.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:22.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:22.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:22.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:51:22.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:22.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10370 cs=0 l=1 rev1=1 rx=0x7f63f0058070 tx=0x7f63f004d110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:22.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10370 cs=0 l=1 rev1=1 rx=0x7f63f0058070 tx=0x7f63f004d110).stop 2021-03-09T09:51:22.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:22.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:22.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:22.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:22.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:51:22.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:22.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:22.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:22.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:22.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:22.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:22.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:22.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022180 con 0x7f63d8003400 2021-03-09T09:51:22.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:22.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bcb0 con 0x7f63d8008840 2021-03-09T09:51:22.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:22.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d820 con 0x7f63e0018390 2021-03-09T09:51:22.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:22.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:22.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:22.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:22.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:51:22.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:22.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:22.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:22.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:22.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:22.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:22.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:22.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:22.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:22.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:22.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:22.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:22.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:51:22.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:51:22.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:22.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:22.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065a30 con 0x7f63e0018390 2021-03-09T09:51:22.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:22.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:22.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.634+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012120 con 0x7f63e0018390 2021-03-09T09:51:22.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.635+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:51:22.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.635+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:22.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.635+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10384 cs=0 l=1 rev1=1 rx=0x7f63f4048d30 tx=0x7f63f4063250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:22.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.635+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:51:22.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.635+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:22.649 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:22.639+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28476 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:22.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400d1c0 con 0x7f63e0018390 2021-03-09T09:51:22.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.639+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:22.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.639+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:22.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.639+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:22.651 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:22.651 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:22.651 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:22.652 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:22.652 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:22.652 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:22.652 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:22.652 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:22.653 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:22.653 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:22.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400e8d0 con 0x7f63e0018390 2021-03-09T09:51:22.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.639+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:22.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.639+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:51:22.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:22.640+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0018390 2021-03-09T09:51:23.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:23.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:23.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:23.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:24.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:24.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:24.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:25.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:25.071+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:25.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:25.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:25.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:25.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:26.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:26.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:26.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:26.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:27.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:27.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:27.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:27.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:27.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10384 cs=0 l=1 rev1=1 rx=0x7f63f4048d30 tx=0x7f63f4063250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:27.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10384 cs=0 l=1 rev1=1 rx=0x7f63f4048d30 tx=0x7f63f4063250).stop 2021-03-09T09:51:27.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:27.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:27.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:27.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:27.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:51:27.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:27.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:27.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:27.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:27.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:27.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:27.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:27.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:51:27.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:27.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022180 con 0x7f63d8008840 2021-03-09T09:51:27.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:27.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002eeb0 con 0x7f6400146050 2021-03-09T09:51:27.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:27.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:27.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:27.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:27.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:27.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:27.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:51:27.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:27.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:27.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:27.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:27.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.640+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:27.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:27.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:27.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:27.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:27.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:27.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:51:27.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:51:27.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:27.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:27.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012120 con 0x7f6400146050 2021-03-09T09:51:27.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:27.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:27.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065a30 con 0x7f6400146050 2021-03-09T09:51:27.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:51:27.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:27.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10400 cs=0 l=1 rev1=1 rx=0x7f63f4053ff0 tx=0x7f63f400e8d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:27.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:51:27.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.641+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:27.656 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:27.643+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28516 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:27.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404cbc0 con 0x7f6400146050 2021-03-09T09:51:27.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.644+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:27.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.644+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:27.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.644+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:27.658 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:27.658 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:27.658 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:27.659 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:27.659 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:27.659 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:27.659 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:27.659 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:27.660 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:27.660 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:27.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.649+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009140 con 0x7f6400146050 2021-03-09T09:51:27.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.649+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:27.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.649+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:51:27.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:27.649+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404cbc0 con 0x7f6400146050 2021-03-09T09:51:28.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:28.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:28.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:28.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:28.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:28.571+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:29.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:29.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:29.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:29.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:30.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:30.072+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:30.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:30.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:30.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:30.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:31.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:31.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:31.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:31.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:32.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:32.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:32.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:32.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:32.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10400 cs=0 l=1 rev1=1 rx=0x7f63f4053ff0 tx=0x7f63f400e8d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:32.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10400 cs=0 l=1 rev1=1 rx=0x7f63f4053ff0 tx=0x7f63f400e8d0).stop 2021-03-09T09:51:32.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:32.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:32.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:32.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:32.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:51:32.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:32.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:32.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:32.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:32.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:32.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:32.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:32.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f0b0 con 0x7f63d8003400 2021-03-09T09:51:32.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:32.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016080 con 0x7f63d8008840 2021-03-09T09:51:32.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:32.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d820 con 0x7f63e0018390 2021-03-09T09:51:32.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:32.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:32.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:32.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:32.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:32.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.644+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:32.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:51:32.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:32.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:32.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:32.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:32.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:32.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:32.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:32.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:32.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:32.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:32.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:51:32.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:51:32.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:32.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:32.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015510 con 0x7f63e0018390 2021-03-09T09:51:32.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:32.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:32.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:51:32.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:51:32.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.645+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:32.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.647+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10413 cs=0 l=1 rev1=1 rx=0x7f63fc013970 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:32.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.647+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:51:32.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.647+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:32.660 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:32.647+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28551 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:32.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.649+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f63e0018390 2021-03-09T09:51:32.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.649+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:32.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.649+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:32.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.649+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:32.661 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:32.662 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:32.662 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:32.662 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:32.662 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:32.663 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:32.663 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:32.663 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:32.663 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:32.663 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:32.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.651+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f63e0018390 2021-03-09T09:51:32.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.651+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:32.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.651+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:51:32.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:32.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0018390 2021-03-09T09:51:33.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:33.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:33.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:33.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:34.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:34.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:34.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:34.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:35.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:35.073+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:35.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:35.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:35.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:36.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:36.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:36.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:36.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:37.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:37.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:37.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:37.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:37.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10413 cs=0 l=1 rev1=1 rx=0x7f63fc013970 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:37.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10413 cs=0 l=1 rev1=1 rx=0x7f63fc013970 tx=0x7f63fc009a00).stop 2021-03-09T09:51:37.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:37.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:37.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:37.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:37.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:51:37.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:37.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:37.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:37.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:37.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:37.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:37.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:37.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:51:37.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:37.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b8f0 con 0x7f63d8008840 2021-03-09T09:51:37.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:37.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00292a0 con 0x7f6400146050 2021-03-09T09:51:37.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:37.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:37.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.648+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:37.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.649+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:37.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.649+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:37.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.649+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:37.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.649+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:51:37.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.649+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:37.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.649+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:37.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.649+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:37.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.649+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:37.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.649+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:37.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:37.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:37.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:37.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:37.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:37.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:51:37.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:51:37.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:37.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:37.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bc60 con 0x7f6400146050 2021-03-09T09:51:37.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:37.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:37.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012120 con 0x7f6400146050 2021-03-09T09:51:37.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:51:37.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:37.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:51:37.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10426 cs=0 l=1 rev1=1 rx=0x7f63f4033040 tx=0x7f63f4005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:37.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.650+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:37.666 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:37.652+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28591 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:37.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.653+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400e770 con 0x7f6400146050 2021-03-09T09:51:37.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.653+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:37.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.653+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:37.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.653+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:37.668 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:37.668 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:37.668 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:37.669 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:37.669 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:37.669 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:37.669 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:37.670 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:37.670 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:37.670 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:37.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.657+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f6400146050 2021-03-09T09:51:37.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.657+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:37.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.657+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:51:37.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:37.657+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402ce10 con 0x7f6400146050 2021-03-09T09:51:38.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:38.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:38.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:38.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:38.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:38.573+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:39.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:39.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:39.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:39.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:40.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:40.074+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:40.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:40.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:40.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:40.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:41.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:41.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:41.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:41.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:42.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:42.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:42.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:42.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:42.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10426 cs=0 l=1 rev1=1 rx=0x7f63f4033040 tx=0x7f63f4005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:42.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10426 cs=0 l=1 rev1=1 rx=0x7f63f4033040 tx=0x7f63f4005110).stop 2021-03-09T09:51:42.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:42.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:42.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:42.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:42.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:51:42.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:42.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:42.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:42.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:42.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:42.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:42.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:42.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00691f0 con 0x7f63d8003400 2021-03-09T09:51:42.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:42.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e8b0 con 0x7f63d8008840 2021-03-09T09:51:42.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:42.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011f40 con 0x7f63e0018390 2021-03-09T09:51:42.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:42.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:42.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:42.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:42.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:51:42.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:42.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:42.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:42.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:42.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:42.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:42.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.653+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:42.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:42.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:42.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:42.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:42.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:42.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:51:42.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:51:42.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:42.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:42.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026700 con 0x7f63e0018390 2021-03-09T09:51:42.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:42.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:42.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bc60 con 0x7f63e0018390 2021-03-09T09:51:42.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:51:42.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.654+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:42.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.655+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:51:42.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.655+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:42.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.655+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10440 cs=0 l=1 rev1=1 rx=0x7f63f4009960 tx=0x7f63f4048cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:42.668 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:42.656+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28630 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:42.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404c520 con 0x7f63e0018390 2021-03-09T09:51:42.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.658+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:42.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.658+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:42.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.658+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:42.669 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:42.670 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:42.670 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:42.670 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:42.670 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:42.671 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:42.671 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:42.671 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:42.671 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:42.671 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:42.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404c520 con 0x7f63e0018390 2021-03-09T09:51:42.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.661+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:42.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.661+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:51:42.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:42.663+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T09:51:43.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:43.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:43.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:43.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:44.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:44.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:44.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:44.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:45.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:45.075+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:45.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:45.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:45.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:45.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:46.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:46.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:46.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:46.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:47.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:47.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:47.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:47.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:47.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10440 cs=0 l=1 rev1=1 rx=0x7f63f4009960 tx=0x7f63f4048cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:47.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10440 cs=0 l=1 rev1=1 rx=0x7f63f4009960 tx=0x7f63f4048cf0).stop 2021-03-09T09:51:47.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:47.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:47.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:47.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:47.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:51:47.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:47.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:47.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:47.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:47.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:47.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:47.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:47.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016080 con 0x7f63d8003400 2021-03-09T09:51:47.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:47.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00294a0 con 0x7f63d8008840 2021-03-09T09:51:47.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:47.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:51:47.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.658+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:47.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.659+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:47.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.659+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:47.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.659+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:47.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.659+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:47.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.659+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:47.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.659+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:51:47.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.659+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:47.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.659+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:47.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.660+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:47.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.660+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:47.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.660+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:47.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:47.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:47.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:47.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:47.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:47.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:51:47.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:51:47.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:47.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:47.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000bc00 con 0x7f6400146050 2021-03-09T09:51:47.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:47.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:47.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026700 con 0x7f6400146050 2021-03-09T09:51:47.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:51:47.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.661+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:47.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.662+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10461 cs=0 l=1 rev1=1 rx=0x7f63f4072860 tx=0x7f63f404c520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:47.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.662+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:51:47.672 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:47.662+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28670 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:47.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.662+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:47.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.663+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404b270 con 0x7f6400146050 2021-03-09T09:51:47.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.663+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:47.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.663+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:47.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.663+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:47.676 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:47.676 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:47.676 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:47.676 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:47.676 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:47.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.666+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404b270 con 0x7f6400146050 2021-03-09T09:51:47.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.666+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:47.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.666+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:51:47.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:47.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f6400146050 2021-03-09T09:51:48.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:48.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:48.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:48.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:48.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:48.574+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:49.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:49.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:49.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:49.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:50.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:50.076+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:50.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:50.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:50.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:50.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:51.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:51.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:51.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:51.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:52.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:52.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:52.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:51:52.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:52.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10461 cs=0 l=1 rev1=1 rx=0x7f63f4072860 tx=0x7f63f404c520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:52.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10461 cs=0 l=1 rev1=1 rx=0x7f63f4072860 tx=0x7f63f404c520).stop 2021-03-09T09:51:52.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:52.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:52.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:52.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:52.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:51:52.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:52.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:52.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:52.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:52.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:52.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:52.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:52.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b8f0 con 0x7f63d8003400 2021-03-09T09:51:52.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:52.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00691f0 con 0x7f63d8008840 2021-03-09T09:51:52.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:52.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e0018390 2021-03-09T09:51:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:51:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:52.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.663+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.664+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.664+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.664+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:52.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:52.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:51:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:51:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:51:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013280 con 0x7f63e0018390 2021-03-09T09:51:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:51:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.665+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.666+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:51:52.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.666+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:52.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.666+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10472 cs=0 l=1 rev1=1 rx=0x7f63e0068880 tx=0x7f63f005dfb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:52.677 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:52.667+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28707 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63e0018390 2021-03-09T09:51:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.668+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.668+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.668+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:52.680 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:52.681 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T09:51:52.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.668+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:52.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.668+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:51:52.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:52.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e0018390 2021-03-09T09:51:53.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:53.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:53.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:53.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:54.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:54.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:54.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:54.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:55.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:55.077+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:51:55.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:55.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:55.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:55.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:56.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:56.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:56.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:56.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:57.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:57.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:57.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:51:57.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:51:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10472 cs=0 l=1 rev1=1 rx=0x7f63e0068880 tx=0x7f63f005dfb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:51:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10472 cs=0 l=1 rev1=1 rx=0x7f63e0068880 tx=0x7f63f005dfb0).stop 2021-03-09T09:51:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:51:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:51:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:51:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:51:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:51:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:51:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:51:57.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:51:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011f40 con 0x7f63d8003400 2021-03-09T09:51:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00294a0 con 0x7f63d8008840 2021-03-09T09:51:57.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:51:57.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c4d0 con 0x7f6400146050 2021-03-09T09:51:57.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:51:57.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:51:57.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.668+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:57.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.669+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:51:57.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.669+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:51:57.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.669+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:51:57.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.669+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:51:57.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.669+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:51:57.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.669+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:51:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.670+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:51:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.670+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:51:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.670+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:51:57.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:51:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:51:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:51:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:51:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:51:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:57.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a860 con 0x7f6400146050 2021-03-09T09:51:57.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:51:57.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:51:57.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:51:57.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:51:57.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.671+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:57.682 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:51:57.672+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28747 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:51:57.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.672+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10488 cs=0 l=1 rev1=1 rx=0x7f63f0054010 tx=0x7f63f0036d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:51:57.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.672+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:51:57.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.672+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:51:57.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00038c0 con 0x7f6400146050 2021-03-09T09:51:57.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.673+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:51:57.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.673+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:51:57.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.673+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:51:57.685 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:51:57.685 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:51:57.685 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:57.685 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:51:57.685 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:51:57.686 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:51:57.686 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:51:57.686 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:51:57.686 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:51:57.687 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:51:57.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003b480 con 0x7f6400146050 2021-03-09T09:51:57.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.677+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:51:57.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.677+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:51:57.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:57.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00038c0 con 0x7f6400146050 2021-03-09T09:51:58.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:58.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:58.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:58.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:51:58.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:58.575+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:51:59.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:59.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:51:59.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:51:59.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:00.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:00.078+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:00.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:00.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:00.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:00.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:01.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:01.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:01.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:02.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:02.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:02.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:02.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:02.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10488 cs=0 l=1 rev1=1 rx=0x7f63f0054010 tx=0x7f63f0036d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:02.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10488 cs=0 l=1 rev1=1 rx=0x7f63f0054010 tx=0x7f63f0036d90).stop 2021-03-09T09:52:02.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:02.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:02.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:02.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:02.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:52:02.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:02.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:02.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:02.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.672+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:02.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:02.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:02.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:02.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e8b0 con 0x7f63d8003400 2021-03-09T09:52:02.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:02.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023590 con 0x7f63d8008840 2021-03-09T09:52:02.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:02.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63e0018390 2021-03-09T09:52:02.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:02.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:02.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:02.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:02.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.673+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:02.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.674+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:02.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.674+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:52:02.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.674+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:02.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.674+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:02.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.674+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:02.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.674+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:02.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.674+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:02.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:02.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:02.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:02.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:02.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:02.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:52:02.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:52:02.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:02.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:02.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0000ae0 con 0x7f63e0018390 2021-03-09T09:52:02.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:02.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:02.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015510 con 0x7f63e0018390 2021-03-09T09:52:02.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:52:02.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.675+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:02.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.677+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:52:02.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.677+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:02.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.677+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10504 cs=0 l=1 rev1=1 rx=0x7f63e001c020 tx=0x7f63fc03ff80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:02.687 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:02.677+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28784 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071120 con 0x7f63e0018390 2021-03-09T09:52:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.677+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.677+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.677+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:02.689 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:02.689 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:02.689 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:02.689 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:02.689 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:02.690 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:02.690 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:02.690 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:02.690 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:02.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071120 con 0x7f63e0018390 2021-03-09T09:52:02.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.679+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:02.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.680+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:52:02.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:02.681+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0018390 2021-03-09T09:52:03.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:03.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:03.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:03.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:04.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:04.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:04.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:04.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:05.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:05.079+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:05.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:05.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:05.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:05.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:06.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:06.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:07.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:07.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.678+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.678+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.678+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10504 cs=0 l=1 rev1=1 rx=0x7f63e001c020 tx=0x7f63fc03ff80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10504 cs=0 l=1 rev1=1 rx=0x7f63e001c020 tx=0x7f63fc03ff80).stop 2021-03-09T09:52:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:07.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:52:07.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:07.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:07.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:07.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:07.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:07.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:07.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:07.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T09:52:07.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b8f0 con 0x7f63d8008840 2021-03-09T09:52:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:52:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.679+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:07.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:07.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:52:07.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:07.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:07.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:07.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:07.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:07.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:07.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:07.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.680+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:07.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:07.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:07.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:52:07.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:52:07.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:07.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:07.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:52:07.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:07.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:07.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000bc00 con 0x7f6400146050 2021-03-09T09:52:07.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:52:07.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:07.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10515 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:07.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:52:07.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:07.693 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:07.681+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28824 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:07.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.683+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400e740 con 0x7f6400146050 2021-03-09T09:52:07.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.683+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:07.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.683+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:07.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.683+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:07.695 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:07.695 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:07.695 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:07.695 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:07.696 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:07.696 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:07.696 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:07.696 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:07.697 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:07.697 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:07.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.684+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400e740 con 0x7f6400146050 2021-03-09T09:52:07.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.685+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:07.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.685+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:52:07.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:07.686+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404d290 con 0x7f6400146050 2021-03-09T09:52:08.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:08.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:08.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:08.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:08.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:08.577+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:09.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:09.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:09.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:09.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:10.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:10.080+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:10.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:10.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:10.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:10.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:11.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:11.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:11.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:11.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:12.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:12.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:12.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:12.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:12.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10515 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:12.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10515 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4017ed0).stop 2021-03-09T09:52:12.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:12.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:12.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:12.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:12.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:52:12.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:12.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:12.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:12.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:12.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:12.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:12.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:12.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00294a0 con 0x7f63d8003400 2021-03-09T09:52:12.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:12.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069010 con 0x7f63d8008840 2021-03-09T09:52:12.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:12.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000faa0 con 0x7f63e0018390 2021-03-09T09:52:12.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:12.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:12.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:12.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.683+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:52:12.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.683+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:12.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.683+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:12.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.682+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:12.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.683+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:12.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.683+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:12.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.683+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:12.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.683+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:12.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.683+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:12.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:12.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:12.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:12.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:12.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:12.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:52:12.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:52:12.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:12.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:12.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00116b0 con 0x7f63e0018390 2021-03-09T09:52:12.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:12.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:12.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:52:12.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:52:12.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.684+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:12.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.685+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10525 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4072eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:12.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.685+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:52:12.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.685+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:12.698 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:12.686+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28861 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:12.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405c8b0 con 0x7f63e0018390 2021-03-09T09:52:12.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.687+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:12.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.687+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:12.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.687+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:12.700 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:12.700 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:12.701 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:12.701 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:12.701 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:12.701 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:12.701 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:12.702 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:12.702 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:12.702 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:12.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005480 con 0x7f63e0018390 2021-03-09T09:52:12.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.690+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:12.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.690+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:52:12.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:12.691+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405c8b0 con 0x7f63e0018390 2021-03-09T09:52:13.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:13.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:13.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:13.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:14.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:14.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:14.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:14.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:15.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:15.080+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:15.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:15.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:15.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:15.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:16.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:16.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:16.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:16.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:17.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:17.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10525 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4072eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10525 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4072eb0).stop 2021-03-09T09:52:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:17.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:52:17.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:17.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:17.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:17.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:17.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:17.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T09:52:17.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000eeb0 con 0x7f63d8008840 2021-03-09T09:52:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:52:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.685+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:17.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:17.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:17.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:17.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:17.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:52:17.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:17.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:17.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:17.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:17.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.686+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:17.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:17.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:17.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:17.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:17.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:17.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:52:17.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:52:17.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:17.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:17.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012120 con 0x7f6400146050 2021-03-09T09:52:17.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:17.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:17.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0000ae0 con 0x7f6400146050 2021-03-09T09:52:17.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:52:17.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.687+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:17.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.688+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10534 cs=0 l=1 rev1=1 rx=0x7f63e003c420 tx=0x7f63fc006480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:17.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.688+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:52:17.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.688+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:17.702 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:17.689+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28901 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:17.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012750 con 0x7f6400146050 2021-03-09T09:52:17.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.690+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:17.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.690+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:17.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.690+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:17.704 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:17.704 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:17.705 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:17.705 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:17.705 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:17.705 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:17.706 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:17.706 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:17.706 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:17.706 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:17.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.692+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc021460 con 0x7f6400146050 2021-03-09T09:52:17.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.692+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:17.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.692+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:52:17.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:17.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T09:52:18.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:18.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:18.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:18.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:18.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:18.577+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:19.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:19.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:19.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:19.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:20.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:20.081+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:20.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:20.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:20.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:20.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:21.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:21.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:21.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:22.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:22.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:22.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10534 cs=0 l=1 rev1=1 rx=0x7f63e003c420 tx=0x7f63fc006480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:22.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10534 cs=0 l=1 rev1=1 rx=0x7f63e003c420 tx=0x7f63fc006480).stop 2021-03-09T09:52:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:52:22.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:22.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:22.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:22.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:22.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:22.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:22.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:22.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T09:52:22.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:22.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011f40 con 0x7f63d8008840 2021-03-09T09:52:22.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:22.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00294a0 con 0x7f63e0018390 2021-03-09T09:52:22.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:22.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.690+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:22.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:22.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:22.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:22.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:22.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:52:22.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:22.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:22.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:22.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:22.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:22.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:22.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:22.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:22.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:22.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:22.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:52:22.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:52:22.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:22.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:22.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.691+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00691a0 con 0x7f63e0018390 2021-03-09T09:52:22.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.692+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:22.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.692+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:22.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.692+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00116b0 con 0x7f63e0018390 2021-03-09T09:52:22.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.692+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:52:22.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.692+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:22.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.693+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:52:22.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.693+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:22.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.693+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10551 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4005180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:22.706 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:22.694+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28938 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:22.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.695+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401a450 con 0x7f63e0018390 2021-03-09T09:52:22.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.695+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:22.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.695+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:22.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.695+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:22.707 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:22.707 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:22.708 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:22.708 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:22.708 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:22.708 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:22.709 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:22.709 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:22.709 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:22.709 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:22.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.700+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f401a270 con 0x7f63e0018390 2021-03-09T09:52:22.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.700+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:22.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.700+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:52:22.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:22.701+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402d2a0 con 0x7f63e0018390 2021-03-09T09:52:23.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:23.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:23.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:23.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:24.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:24.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:24.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:24.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:25.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:25.082+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:25.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:25.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:25.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:25.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:26.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:26.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:26.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:26.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:27.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:27.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:27.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.694+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.694+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.694+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10551 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4005180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.694+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10551 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4005180).stop 2021-03-09T09:52:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.694+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:52:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:27.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:27.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:27.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:27.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026750 con 0x7f63d8003400 2021-03-09T09:52:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8008840 2021-03-09T09:52:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025cb0 con 0x7f6400146050 2021-03-09T09:52:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:27.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:27.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:52:27.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:27.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.695+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:52:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:52:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:27.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000bc00 con 0x7f6400146050 2021-03-09T09:52:27.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:27.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:27.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012120 con 0x7f6400146050 2021-03-09T09:52:27.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:52:27.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:27.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10562 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:27.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:52:27.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.696+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:27.710 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:27.699+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 28978 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.699+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc051840 con 0x7f6400146050 2021-03-09T09:52:27.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.699+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.699+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.699+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:27.712 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:27.713 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:27.713 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:27.713 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:27.713 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:27.714 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:27.714 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:27.714 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:27.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.699+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051840 con 0x7f6400146050 2021-03-09T09:52:27.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.699+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:27.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.699+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:52:27.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:27.700+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f6400146050 2021-03-09T09:52:28.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:28.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:28.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:28.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:28.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:28.578+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:29.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:29.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:29.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:29.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:30.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:30.083+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:30.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:30.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:30.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:30.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:31.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:31.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:31.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:31.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:32.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:32.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:32.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:32.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:32.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10562 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:32.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10562 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc003a50).stop 2021-03-09T09:52:32.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:32.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:32.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:52:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011bd0 con 0x7f63d8003400 2021-03-09T09:52:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066840 con 0x7f63d8008840 2021-03-09T09:52:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:52:32.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:32.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:32.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.699+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:32.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.700+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:32.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.700+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.700+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.700+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:52:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.700+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:32.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.700+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:32.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.700+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:32.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.700+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:32.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.700+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:32.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:52:32.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:52:32.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:32.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:32.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:52:32.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:32.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:32.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00691a0 con 0x7f63e0018390 2021-03-09T09:52:32.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:52:32.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.701+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:32.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.702+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002a370 secure :-1 s=READY pgs=10576 cs=0 l=1 rev1=1 rx=0x7f63f4034e50 tx=0x7f63f405be40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:32.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.702+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:52:32.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.702+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:32.715 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:32.704+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29015 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:32.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.705+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f406ada0 con 0x7f63e0018390 2021-03-09T09:52:32.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.705+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:32.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.705+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:32.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.705+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:32.716 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:32.717 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:32.717 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:32.717 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:32.717 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:32.718 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:32.718 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:32.718 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:32.718 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:32.719 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:32.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.707+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f406ada0 con 0x7f63e0018390 2021-03-09T09:52:32.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.707+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:32.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.708+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:52:32.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:32.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f406ada0 con 0x7f63e0018390 2021-03-09T09:52:33.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:33.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:33.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:33.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:34.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:34.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:34.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:35.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:35.084+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:35.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:35.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:35.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:35.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:36.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:36.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:36.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:36.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:37.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:37.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e002a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:52:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e002a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002a370 secure :-1 s=READY pgs=10576 cs=0 l=1 rev1=1 rx=0x7f63f4034e50 tx=0x7f63f405be40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002a370 secure :-1 s=READY pgs=10576 cs=0 l=1 rev1=1 rx=0x7f63f4034e50 tx=0x7f63f405be40).stop 2021-03-09T09:52:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e002a370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002a370 unknown :-1 s=CLOSED pgs=10576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:52:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000eeb0 con 0x7f63d8003400 2021-03-09T09:52:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00294a0 con 0x7f63d8008840 2021-03-09T09:52:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:52:37.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:37.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:37.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.704+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:37.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.705+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.705+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.705+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.705+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:52:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.705+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.705+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.705+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.705+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.705+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:37.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:37.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:37.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:37.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:37.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:52:37.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:52:37.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:37.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:37.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:52:37.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:37.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:37.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a860 con 0x7f6400146050 2021-03-09T09:52:37.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:52:37.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.706+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:37.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.707+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10592 cs=0 l=1 rev1=1 rx=0x7f63f0023220 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:37.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.707+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:52:37.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.707+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:37.724 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:37.708+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29055 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:37.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f6400146050 2021-03-09T09:52:37.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.709+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:37.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.709+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:37.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.709+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:37.726 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:37.727 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:37.727 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:37.727 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:37.727 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:37.728 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:37.728 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:37.728 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:37.728 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:37.728 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:37.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f6400146050 2021-03-09T09:52:37.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.709+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:37.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.709+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:52:37.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:37.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:52:38.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:38.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:38.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:38.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:38.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:38.580+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:39.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:39.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:39.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:39.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:40.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:40.085+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:40.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:40.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:40.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:40.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:41.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:41.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:41.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:41.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:42.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:42.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:42.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:42.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:42.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10592 cs=0 l=1 rev1=1 rx=0x7f63f0023220 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:42.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10592 cs=0 l=1 rev1=1 rx=0x7f63f0023220 tx=0x7f63f0013b80).stop 2021-03-09T09:52:42.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:42.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:42.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:42.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:42.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:52:42.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:42.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:42.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:42.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:42.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:42.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:42.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:42.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025cb0 con 0x7f63d8003400 2021-03-09T09:52:42.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011f40 con 0x7f63d8008840 2021-03-09T09:52:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63e0018390 2021-03-09T09:52:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:42.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:42.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:42.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:42.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.709+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:42.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:52:42.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:42.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:42.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:52:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:52:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:42.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:42.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012120 con 0x7f63e0018390 2021-03-09T09:52:42.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:42.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:52:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:52:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.710+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.711+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:52:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.711+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10605 cs=0 l=1 rev1=1 rx=0x7f63f4005150 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:42.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.711+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:42.725 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:42.713+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29092 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:42.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.715+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400e440 con 0x7f63e0018390 2021-03-09T09:52:42.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.715+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:42.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.715+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:42.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.715+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:42.729 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:42.729 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:42.729 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:42.729 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:42.729 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:42.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.716+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400e440 con 0x7f63e0018390 2021-03-09T09:52:42.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.716+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:42.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.716+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:52:42.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:42.717+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402d2a0 con 0x7f63e0018390 2021-03-09T09:52:43.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:43.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:43.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:43.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:44.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:44.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:44.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:44.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:45.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:45.086+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:45.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:45.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:45.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:46.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:46.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:46.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:46.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:47.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:47.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:47.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:47.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:47.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10605 cs=0 l=1 rev1=1 rx=0x7f63f4005150 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:47.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10605 cs=0 l=1 rev1=1 rx=0x7f63f4005150 tx=0x7f63f4017ed0).stop 2021-03-09T09:52:47.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:47.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:47.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:47.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:47.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:52:47.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:47.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:47.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:47.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:47.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:47.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:47.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:47.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:52:47.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:47.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066840 con 0x7f63d8008840 2021-03-09T09:52:47.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:47.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00294a0 con 0x7f6400146050 2021-03-09T09:52:47.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:47.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:47.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:47.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:47.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:47.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:47.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:52:47.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:47.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:47.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.714+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:52:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:52:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026700 con 0x7f6400146050 2021-03-09T09:52:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:47.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:47.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:52:47.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:52:47.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.715+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:47.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.716+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10617 cs=0 l=1 rev1=1 rx=0x7f63f0054010 tx=0x7f63f0030220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:47.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.716+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:52:47.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.716+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:47.728 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:47.718+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29132 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:47.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.719+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028ec0 con 0x7f6400146050 2021-03-09T09:52:47.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.719+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:47.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.719+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:47.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.719+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:47.730 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:47.730 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:47.732 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.723+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028ec0 con 0x7f6400146050 2021-03-09T09:52:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.723+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:47.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.723+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:52:47.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:47.724+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:52:48.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:48.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:48.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:48.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:48.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:48.581+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:49.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:49.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:49.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:49.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:50.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:50.087+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:50.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:50.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:50.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:50.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:51.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:51.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:51.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:51.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:52.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:52.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:52.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:52.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:52.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10617 cs=0 l=1 rev1=1 rx=0x7f63f0054010 tx=0x7f63f0030220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:52.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10617 cs=0 l=1 rev1=1 rx=0x7f63f0054010 tx=0x7f63f0030220).stop 2021-03-09T09:52:52.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:52.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:52.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:52.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:52.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:52:52.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:52.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:52.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:52.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:52.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:52.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:52.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:52.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005edd0 con 0x7f63d8003400 2021-03-09T09:52:52.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:52.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c9e0 con 0x7f63d8008840 2021-03-09T09:52:52.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:52.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d3a0 con 0x7f63e0018390 2021-03-09T09:52:52.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:52.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:52.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.718+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:52.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.719+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:52.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.719+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:52.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.719+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:52.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.719+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:52:52.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.719+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:52.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.719+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:52.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.719+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:52.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.719+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:52.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.719+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:52.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:52.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:52.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:52.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:52.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:52.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:52:52.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:52:52.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:52.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:52.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bf70 con 0x7f63e0018390 2021-03-09T09:52:52.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:52.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:52.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000bc00 con 0x7f63e0018390 2021-03-09T09:52:52.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:52:52.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.720+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:52.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.721+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:52:52.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.721+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:52.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.721+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10634 cs=0 l=1 rev1=1 rx=0x7f63e0016ea0 tx=0x7f63fc006560).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:52.733 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:52.722+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29169 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:52.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.724+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63e0018390 2021-03-09T09:52:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.724+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.724+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.724+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:52.737 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:52.737 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:52.737 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:52.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63e0018390 2021-03-09T09:52:52.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.727+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:52.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.727+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:52:52.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:52.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0018390 2021-03-09T09:52:53.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:53.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:53.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:53.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:54.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:54.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:54.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:54.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:55.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:55.088+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:52:55.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:55.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:55.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:55.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:56.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:56.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:56.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:56.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:57.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:57.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:57.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:52:57.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:52:57.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10634 cs=0 l=1 rev1=1 rx=0x7f63e0016ea0 tx=0x7f63fc006560).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:52:57.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10634 cs=0 l=1 rev1=1 rx=0x7f63e0016ea0 tx=0x7f63fc006560).stop 2021-03-09T09:52:57.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:57.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:52:57.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:52:57.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:52:57.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:52:57.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:57.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:52:57.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:57.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:52:57.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:52:57.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:52:57.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:57.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026010 con 0x7f63d8003400 2021-03-09T09:52:57.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:57.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8008840 2021-03-09T09:52:57.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:52:57.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:52:57.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:52:57.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:52:57.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:57.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:52:57.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:52:57.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:52:57.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:52:57.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:52:57.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:52:57.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:52:57.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:52:57.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.723+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:52:57.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:52:57.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:57.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:57.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:52:57.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:52:57.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:52:57.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:52:57.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:57.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:57.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:52:57.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:52:57.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:52:57.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012120 con 0x7f6400146050 2021-03-09T09:52:57.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:52:57.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:57.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:52:57.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10644 cs=0 l=1 rev1=1 rx=0x7f63f404bf80 tx=0x7f63f4063250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:52:57.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.724+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:52:57.739 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:52:57.727+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29209 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:52:57.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405beb0 con 0x7f6400146050 2021-03-09T09:52:57.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.727+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:52:57.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.727+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:52:57.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.727+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:52:57.742 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:52:57.742 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:52:57.742 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:52:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:52:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:52:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:52:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:52:57.744 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:52:57.744 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:52:57.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405beb0 con 0x7f6400146050 2021-03-09T09:52:57.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.727+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:52:57.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.727+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:52:57.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:57.729+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404d290 con 0x7f6400146050 2021-03-09T09:52:58.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:58.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:58.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:58.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:52:58.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:58.582+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:52:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:59.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:52:59.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:52:59.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:00.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:00.089+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:00.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:00.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:00.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:00.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:01.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:01.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:01.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:01.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:02.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:02.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:02.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.727+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:02.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.727+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:02.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.727+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10644 cs=0 l=1 rev1=1 rx=0x7f63f404bf80 tx=0x7f63f4063250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:02.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.727+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10644 cs=0 l=1 rev1=1 rx=0x7f63f404bf80 tx=0x7f63f4063250).stop 2021-03-09T09:53:02.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.727+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:02.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:02.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:02.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:02.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:53:02.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:02.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:02.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:02.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:02.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:02.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:02.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:02.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017790 con 0x7f63d8003400 2021-03-09T09:53:02.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:02.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00265c0 con 0x7f63d8008840 2021-03-09T09:53:02.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:02.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028bc0 con 0x7f63e0018390 2021-03-09T09:53:02.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:02.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:02.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:02.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:02.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.728+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:02.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.729+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:02.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.729+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:53:02.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.729+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:02.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.729+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:02.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.729+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:02.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.729+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:02.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.729+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:02.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:02.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:02.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:02.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:02.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:02.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:53:02.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:53:02.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:02.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:02.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e090 con 0x7f63e0018390 2021-03-09T09:53:02.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:02.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:02.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026700 con 0x7f63e0018390 2021-03-09T09:53:02.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:53:02.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.730+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:02.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.732+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10665 cs=0 l=1 rev1=1 rx=0x7f63e00291a0 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:02.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.732+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:53:02.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.732+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:02.744 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:02.732+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29246 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:02.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00312f0 con 0x7f63e0018390 2021-03-09T09:53:02.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.733+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:02.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.733+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:02.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.733+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:02.746 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:02.746 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:02.747 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:02.747 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:02.747 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:02.747 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:02.747 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:02.748 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:02.748 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:02.748 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:02.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.736+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003b480 con 0x7f63e0018390 2021-03-09T09:53:02.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.736+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:02.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.736+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:53:02.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:02.737+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00312f0 con 0x7f63e0018390 2021-03-09T09:53:03.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:03.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:03.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:03.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:04.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:04.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:04.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:04.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:05.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:05.090+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:05.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:05.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:05.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:05.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:06.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:06.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:07.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:07.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:07.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:07.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:07.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10665 cs=0 l=1 rev1=1 rx=0x7f63e00291a0 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:07.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10665 cs=0 l=1 rev1=1 rx=0x7f63e00291a0 tx=0x7f63f004de00).stop 2021-03-09T09:53:07.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:07.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:07.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:07.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:07.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:53:07.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:07.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:07.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:07.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:07.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:07.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:07.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:07.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00095d0 con 0x7f63d8003400 2021-03-09T09:53:07.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:07.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8008840 2021-03-09T09:53:07.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:07.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013ee0 con 0x7f6400146050 2021-03-09T09:53:07.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:07.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:07.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:07.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:07.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.733+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:53:07.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:07.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:07.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:07.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:07.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:07.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:07.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T09:53:07.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:07.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:07.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.734+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:07.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:07.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:07.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:53:07.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:53:07.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:07.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:07.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0021740 con 0x7f6400146050 2021-03-09T09:53:07.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:07.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:07.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:53:07.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:53:07.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:07.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:53:07.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:07.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.735+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10673 cs=0 l=1 rev1=1 rx=0x7f63f404bfd0 tx=0x7f63f400e2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:07.751 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:07.737+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29286 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:07.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f406a910 con 0x7f6400146050 2021-03-09T09:53:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.738+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.738+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.738+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:07.752 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:07.753 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:07.754 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:07.755 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:07.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f406a6e0 con 0x7f6400146050 2021-03-09T09:53:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.738+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.738+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:53:07.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:07.739+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404d290 con 0x7f6400146050 2021-03-09T09:53:08.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:08.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:08.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:08.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:08.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:08.584+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:09.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:09.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:09.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:09.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:10.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:10.091+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:10.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:10.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:10.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:10.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:11.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:11.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:11.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:11.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:12.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:12.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:12.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:12.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:12.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10673 cs=0 l=1 rev1=1 rx=0x7f63f404bfd0 tx=0x7f63f400e2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:12.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10673 cs=0 l=1 rev1=1 rx=0x7f63f404bfd0 tx=0x7f63f400e2d0).stop 2021-03-09T09:53:12.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:12.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:12.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:12.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:12.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:53:12.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:12.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:12.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:12.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:12.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028bc0 con 0x7f63d8003400 2021-03-09T09:53:12.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:12.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d5a0 con 0x7f63d8008840 2021-03-09T09:53:12.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:12.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:53:12.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:12.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:12.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.737+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:12.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.738+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:12.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.738+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.738+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.738+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:53:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.738+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:12.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.738+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:12.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.739+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:12.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.739+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:12.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.739+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:12.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:12.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:12.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:12.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:12.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:12.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:53:12.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:53:12.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:12.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:12.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000eb10 con 0x7f63e0018390 2021-03-09T09:53:12.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:12.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:12.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0021740 con 0x7f63e0018390 2021-03-09T09:53:12.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:53:12.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:12.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10690 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f4034e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:12.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:53:12.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.740+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:12.757 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:12.741+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29323 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:12.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034b80 con 0x7f63e0018390 2021-03-09T09:53:12.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.742+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:12.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.742+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:12.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.742+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:12.759 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:12.761 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:12.761 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:12.761 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:12.761 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:12.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034b80 con 0x7f63e0018390 2021-03-09T09:53:12.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.743+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:12.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.743+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:53:12.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:12.744+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022430 con 0x7f63e0018390 2021-03-09T09:53:13.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:13.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:13.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:13.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:14.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:14.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:14.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:14.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:15.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:15.092+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:15.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:15.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:15.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:15.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:16.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:16.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:16.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:16.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:17.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:17.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:17.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:17.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:17.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10690 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f4034e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:17.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10690 cs=0 l=1 rev1=1 rx=0x7f63f4000db0 tx=0x7f63f4034e00).stop 2021-03-09T09:53:17.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:17.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:17.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:17.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:17.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:53:17.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:17.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:17.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:17.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:17.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:17.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8003400 2021-03-09T09:53:17.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:17.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002abb0 con 0x7f63d8008840 2021-03-09T09:53:17.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:17.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000faa0 con 0x7f6400146050 2021-03-09T09:53:17.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:17.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:17.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:17.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.742+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:17.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.743+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:17.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.743+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:53:17.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.743+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:17.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.743+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:17.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.743+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:17.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.743+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:17.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.743+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:17.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:17.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:17.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:17.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:17.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:17.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:53:17.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:53:17.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:17.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:17.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e350 con 0x7f6400146050 2021-03-09T09:53:17.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:17.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:17.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000eb10 con 0x7f6400146050 2021-03-09T09:53:17.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:53:17.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:17.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10702 cs=0 l=1 rev1=1 rx=0x7f63f4039050 tx=0x7f63f406b370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:17.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:53:17.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.744+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:17.776 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:17.746+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29363 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:17.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b400 con 0x7f6400146050 2021-03-09T09:53:17.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.747+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:17.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.747+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:17.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.747+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:17.779 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:17.780 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:17.780 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:17.781 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:17.781 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:17.782 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:17.782 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:17.783 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:17.783 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:17.784 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:17.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f6400146050 2021-03-09T09:53:17.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.747+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:17.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.747+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:53:17.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:17.748+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022430 con 0x7f6400146050 2021-03-09T09:53:18.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:18.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:18.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:18.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:18.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:18.585+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:19.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:19.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:19.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:19.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:20.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:20.093+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:20.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:20.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:21.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:21.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:21.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:21.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:22.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:22.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:22.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10702 cs=0 l=1 rev1=1 rx=0x7f63f4039050 tx=0x7f63f406b370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:22.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10702 cs=0 l=1 rev1=1 rx=0x7f63f4039050 tx=0x7f63f406b370).stop 2021-03-09T09:53:22.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:22.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:22.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:22.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:22.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:53:22.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00257b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:22.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:22.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:22.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:22.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:22.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:22.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:22.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b500 con 0x7f63d8003400 2021-03-09T09:53:22.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:22.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:53:22.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:22.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63e0018390 2021-03-09T09:53:22.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.746+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:22.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.747+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:22.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.747+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:22.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.747+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:22.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.747+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:22.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.747+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00257b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:22.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.747+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:53:22.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.747+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:22.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.747+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:22.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.748+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:22.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.748+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:22.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.748+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:22.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.749+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:22.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.749+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:22.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.749+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:22.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.749+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:22.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.749+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:22.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:53:22.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:53:22.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:22.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:22.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:53:22.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:22.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:22.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e350 con 0x7f63e0018390 2021-03-09T09:53:22.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:53:22.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:22.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00257b0 secure :-1 s=READY pgs=10722 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4038fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:22.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:53:22.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.750+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:22.763 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:22.751+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29400 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:22.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4046b40 con 0x7f63e0018390 2021-03-09T09:53:22.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.751+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:22.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.751+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:22.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.751+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:22.765 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:22.765 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:22.766 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:22.766 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:22.766 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:22.766 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:22.767 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:22.767 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:22.767 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:22.767 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:22.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.752+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046b40 con 0x7f63e0018390 2021-03-09T09:53:22.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.752+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:22.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.752+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:53:22.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:22.753+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40488a0 con 0x7f63e0018390 2021-03-09T09:53:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:23.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:23.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:23.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:24.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:24.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:24.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:24.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:25.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:25.094+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:25.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:25.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:25.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:25.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:26.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:26.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:26.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:26.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:27.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00257b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:27.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00257b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:27.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00257b0 secure :-1 s=READY pgs=10722 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4038fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:27.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00257b0 secure :-1 s=READY pgs=10722 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4038fe0).stop 2021-03-09T09:53:27.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:27.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:27.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00257b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:27.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00257b0 unknown :-1 s=CLOSED pgs=10722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:27.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:53:27.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:27.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:27.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:27.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:27.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:27.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:27.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:27.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017790 con 0x7f63d8003400 2021-03-09T09:53:27.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:27.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026ef0 con 0x7f63d8008840 2021-03-09T09:53:27.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:27.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000faa0 con 0x7f6400146050 2021-03-09T09:53:27.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:27.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:27.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.751+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:27.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.752+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:27.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.752+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:27.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.752+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:27.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.752+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:53:27.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.752+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:27.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.752+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:27.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.752+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:27.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.752+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:27.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.752+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:27.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:27.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:27.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:27.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:27.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:27.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:53:27.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:53:27.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:27.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:27.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a860 con 0x7f6400146050 2021-03-09T09:53:27.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:27.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:27.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e090 con 0x7f6400146050 2021-03-09T09:53:27.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:53:27.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.753+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:27.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.754+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:53:27.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.754+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:27.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.754+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10728 cs=0 l=1 rev1=1 rx=0x7f63e0028740 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:27.768 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:27.755+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29440 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:27.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038470 con 0x7f6400146050 2021-03-09T09:53:27.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.756+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:27.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.756+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:27.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.756+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:27.770 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:27.770 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:27.770 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:27.771 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:27.771 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:27.771 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:27.772 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:27.772 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:27.772 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:27.772 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:27.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:53:27.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.758+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:27.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.758+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:53:27.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:27.759+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:53:28.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:28.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:28.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:28.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:28.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:28.587+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:29.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:29.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:29.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:29.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:30.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:30.095+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:30.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:30.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:30.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:30.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:31.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:31.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:31.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:31.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:32.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:32.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:32.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:32.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:32.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10728 cs=0 l=1 rev1=1 rx=0x7f63e0028740 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:32.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10728 cs=0 l=1 rev1=1 rx=0x7f63e0028740 tx=0x7f63f0000bc0).stop 2021-03-09T09:53:32.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:32.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:32.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:32.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:32.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:53:32.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:32.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:32.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0023bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:32.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:32.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:32.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:32.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:32.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:53:32.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:32.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:53:32.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:32.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63e0018390 2021-03-09T09:53:32.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:32.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:32.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:32.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:32.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.756+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:32.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.757+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0023bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:32.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.757+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:53:32.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.757+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:32.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.757+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:32.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:32.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:32.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:32.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:32.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:32.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:32.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:32.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:32.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:53:32.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:53:32.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:32.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:32.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066500 con 0x7f63e0018390 2021-03-09T09:53:32.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:32.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:32.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bf70 con 0x7f63e0018390 2021-03-09T09:53:32.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:53:32.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.758+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:32.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.759+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0023bb0 secure :-1 s=READY pgs=10743 cs=0 l=1 rev1=1 rx=0x7f63e00291d0 tx=0x7f63fc04bd20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:32.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.759+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:53:32.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.759+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:32.772 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:32.760+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29477 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:32.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.761+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f63e0018390 2021-03-09T09:53:32.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.761+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:32.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.761+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:32.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.761+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:32.774 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:32.774 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:32.774 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:32.774 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:32.775 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:32.775 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:32.775 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:32.775 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:32.775 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:32.776 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:32.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.765+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f63e0018390 2021-03-09T09:53:32.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.765+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:32.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.765+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:53:32.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:32.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T09:53:33.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:33.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:33.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:33.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:34.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:34.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:34.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:34.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:35.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:35.096+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:35.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:35.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:35.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:35.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:36.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:36.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:36.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:36.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:37.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:37.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:37.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0023bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:37.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0023bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:37.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0023bb0 secure :-1 s=READY pgs=10743 cs=0 l=1 rev1=1 rx=0x7f63e00291d0 tx=0x7f63fc04bd20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:37.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0023bb0 secure :-1 s=READY pgs=10743 cs=0 l=1 rev1=1 rx=0x7f63e00291d0 tx=0x7f63fc04bd20).stop 2021-03-09T09:53:37.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:37.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:37.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0023bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:37.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0023bb0 unknown :-1 s=CLOSED pgs=10743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:37.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:53:37.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:37.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:37.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:37.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:37.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:37.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:37.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:37.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8003400 2021-03-09T09:53:37.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:37.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:53:37.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:37.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022ff0 con 0x7f6400146050 2021-03-09T09:53:37.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:37.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:37.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:37.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:37.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.761+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:37.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.762+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:37.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.762+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:53:37.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.762+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:37.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.762+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:37.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.763+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:37.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.763+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:37.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.763+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:37.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:37.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:37.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:37.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:37.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:37.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:53:37.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:53:37.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:37.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:37.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017790 con 0x7f6400146050 2021-03-09T09:53:37.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:37.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:37.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a860 con 0x7f6400146050 2021-03-09T09:53:37.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:53:37.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:37.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10762 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f007af80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:37.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.766+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:53:37.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.766+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:37.794 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:37.766+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29517 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:37.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.767+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:53:37.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.767+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:37.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.767+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:37.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.767+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:37.796 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:37.796 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:37.796 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:37.797 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:37.797 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:37.797 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:37.797 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:37.798 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:37.798 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:37.798 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:37.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:53:37.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.771+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:37.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.771+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:53:37.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:37.772+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:53:38.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:38.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:38.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:38.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:38.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:38.587+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:39.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:39.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:39.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:39.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:40.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:40.097+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:40.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:40.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:40.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:40.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:41.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:41.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:41.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:41.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:42.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:42.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:42.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:53:42.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:42.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10762 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f007af80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:42.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10762 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f007af80).stop 2021-03-09T09:53:42.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:42.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:42.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:42.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:42.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:53:42.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:42.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:42.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:42.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:42.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:42.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:42.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:42.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027f00 con 0x7f63d8003400 2021-03-09T09:53:42.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:42.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:53:42.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:42.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002abb0 con 0x7f63e0018390 2021-03-09T09:53:42.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:42.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:42.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.766+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:42.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.767+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:42.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.767+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:42.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.767+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:42.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.767+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:53:42.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.767+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:42.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.767+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:42.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.768+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:42.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.768+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:42.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.768+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:42.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:42.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:42.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:42.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:53:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:53:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e350 con 0x7f63e0018390 2021-03-09T09:53:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:42.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:53:42.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:53:42.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.769+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:42.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.770+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10777 cs=0 l=1 rev1=1 rx=0x7f63f4046fb0 tx=0x7f63f40701e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:42.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.770+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:53:42.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.770+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:42.782 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:42.770+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29554 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:42.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:53:42.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.771+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:42.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.771+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:42.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.771+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:42.784 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:42.784 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:42.785 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:42.785 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:42.785 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:42.785 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:42.785 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:42.786 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:42.786 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:42.786 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:42.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.772+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:53:42.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.772+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:42.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.772+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:53:42.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:42.773+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63e0018390 2021-03-09T09:53:43.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:43.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:43.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:43.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:44.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:44.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:44.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:44.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:45.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:45.098+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:45.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:45.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:45.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:46.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:46.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:46.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:46.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:47.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:47.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:47.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:47.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:47.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10777 cs=0 l=1 rev1=1 rx=0x7f63f4046fb0 tx=0x7f63f40701e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:47.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10777 cs=0 l=1 rev1=1 rx=0x7f63f4046fb0 tx=0x7f63f40701e0).stop 2021-03-09T09:53:47.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:47.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:47.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:47.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:47.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:53:47.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:47.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:47.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:47.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:47.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:47.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:47.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:47.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T09:53:47.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:47.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068710 con 0x7f63d8008840 2021-03-09T09:53:47.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:47.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bb20 con 0x7f6400146050 2021-03-09T09:53:47.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:47.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:47.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:47.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:47.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:47.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:47.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:53:47.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:47.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:47.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:47.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:47.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.772+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:47.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:47.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:47.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:47.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:47.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:47.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:53:47.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:53:47.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:47.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:47.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:53:47.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:47.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:47.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017790 con 0x7f6400146050 2021-03-09T09:53:47.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:53:47.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:47.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:53:47.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:47.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.773+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10781 cs=0 l=1 rev1=1 rx=0x7f63f006c010 tx=0x7f63f0006bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:47.788 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:47.776+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29594 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:47.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005fee0 con 0x7f6400146050 2021-03-09T09:53:47.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.777+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:47.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.777+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:47.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.777+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:47.789 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:47.790 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:47.790 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:47.790 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:47.790 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:47.791 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:47.791 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:47.791 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:47.791 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:47.792 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:47.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005fcb0 con 0x7f6400146050 2021-03-09T09:53:47.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.779+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:47.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.779+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:53:47.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:47.783+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T09:53:48.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:48.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:48.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:48.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:48.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:48.589+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:49.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:49.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:49.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:49.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:50.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:50.099+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:50.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:50.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:50.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:50.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:51.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:51.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:51.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:51.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:52.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:52.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:52.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:53:52.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:52.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10781 cs=0 l=1 rev1=1 rx=0x7f63f006c010 tx=0x7f63f0006bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:52.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10781 cs=0 l=1 rev1=1 rx=0x7f63f006c010 tx=0x7f63f0006bc0).stop 2021-03-09T09:53:52.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:52.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:52.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:52.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:52.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:53:52.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:52.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:52.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:52.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:52.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:52.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:52.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:52.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00290f0 con 0x7f63d8003400 2021-03-09T09:53:52.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:52.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8008840 2021-03-09T09:53:52.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:52.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:53:52.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.776+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:52.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.777+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:52.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.777+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:52.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.777+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:52.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.777+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:52.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.777+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.777+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:53:52.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.777+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.777+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:52.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:52.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:52.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:52.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:53:52.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:53:52.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:52.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:52.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d310 con 0x7f63e0018390 2021-03-09T09:53:52.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:52.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e350 con 0x7f63e0018390 2021-03-09T09:53:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:53:52.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.778+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:52.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.779+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:53:52.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.779+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:52.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.779+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10797 cs=0 l=1 rev1=1 rx=0x7f63f4074070 tx=0x7f63f400d2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:52.804 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:52.781+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29631 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:52.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63e0018390 2021-03-09T09:53:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.782+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:52.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.782+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:52.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.782+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:52.807 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:52.808 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:52.809 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:52.809 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:52.811 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63e0018390 2021-03-09T09:53:52.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.786+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:52.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.786+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:53:52.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:52.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63e0018390 2021-03-09T09:53:53.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:53.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:53.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:53.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:54.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:54.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:54.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:54.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:55.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:55.100+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:53:55.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:55.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:55.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:55.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:56.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:56.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:56.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:56.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:57.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:57.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:57.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:53:57.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:53:57.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10797 cs=0 l=1 rev1=1 rx=0x7f63f4074070 tx=0x7f63f400d2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:53:57.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10797 cs=0 l=1 rev1=1 rx=0x7f63f4074070 tx=0x7f63f400d2a0).stop 2021-03-09T09:53:57.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:57.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:53:57.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:53:57.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:53:57.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:53:57.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:57.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:53:57.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:57.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:53:57.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:53:57.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:53:57.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:57.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T09:53:57.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:57.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00679f0 con 0x7f63d8008840 2021-03-09T09:53:57.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:53:57.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400146050 2021-03-09T09:53:57.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:53:57.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:53:57.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:57.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:53:57.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:53:57.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:53:57.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:53:57.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:53:57.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:53:57.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:53:57.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:53:57.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.782+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:53:57.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:53:57.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:57.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:57.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:53:57.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:53:57.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:53:57.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:53:57.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:57.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:57.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024c50 con 0x7f6400146050 2021-03-09T09:53:57.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:53:57.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:53:57.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066500 con 0x7f6400146050 2021-03-09T09:53:57.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:53:57.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:57.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10810 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc06ff30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:53:57.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.784+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:53:57.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.784+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:53:57.799 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:53:57.786+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29671 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:53:57.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f6400146050 2021-03-09T09:53:57.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.787+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:53:57.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.787+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:53:57.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.787+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:53:57.801 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:53:57.801 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:53:57.801 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:57.802 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:53:57.802 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:53:57.802 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:53:57.802 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:53:57.803 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:53:57.803 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:53:57.803 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:53:57.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f6400146050 2021-03-09T09:53:57.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.787+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:53:57.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.787+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:53:57.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:57.788+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T09:53:58.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:58.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:58.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:58.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:53:58.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:58.590+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:53:59.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:59.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:53:59.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:53:59.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:00.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:00.101+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:00.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:00.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:00.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:00.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:01.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:01.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:01.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:01.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:02.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:02.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:02.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:02.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:02.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10810 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc06ff30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:02.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10810 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc06ff30).stop 2021-03-09T09:54:02.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:02.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:02.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:02.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:54:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:02.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:02.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010cb0 con 0x7f63d8003400 2021-03-09T09:54:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bb20 con 0x7f63d8008840 2021-03-09T09:54:02.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:02.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63e0018390 2021-03-09T09:54:02.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:02.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:02.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:02.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:02.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.787+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:02.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.788+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:02.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.788+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:54:02.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.788+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:02.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.788+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:02.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.788+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:02.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.788+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:02.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.788+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:02.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:02.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:02.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:02.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:54:02.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:54:02.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:02.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:02.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a900 con 0x7f63e0018390 2021-03-09T09:54:02.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:02.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:02.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024c50 con 0x7f63e0018390 2021-03-09T09:54:02.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:54:02.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.789+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:02.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.790+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10827 cs=0 l=1 rev1=1 rx=0x7f63e007c6d0 tx=0x7f63fc0666a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:02.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.790+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:54:02.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.790+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:02.805 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:02.791+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29708 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:02.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0668d0 con 0x7f63e0018390 2021-03-09T09:54:02.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.792+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:02.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.792+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:02.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.792+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:02.806 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:02.806 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:02.807 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:02.807 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:02.807 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:02.807 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:02.808 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:02.808 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:02.808 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:02.808 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:02.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.796+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0668d0 con 0x7f63e0018390 2021-03-09T09:54:02.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.796+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:02.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.796+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:54:02.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:02.797+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e0018390 2021-03-09T09:54:03.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:03.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:03.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:03.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:04.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:04.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:04.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:04.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:05.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:05.102+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:05.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:05.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:05.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:05.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:06.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:06.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:06.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:06.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:07.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:07.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:07.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:07.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:07.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10827 cs=0 l=1 rev1=1 rx=0x7f63e007c6d0 tx=0x7f63fc0666a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:07.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10827 cs=0 l=1 rev1=1 rx=0x7f63e007c6d0 tx=0x7f63fc0666a0).stop 2021-03-09T09:54:07.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:07.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:54:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:07.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:07.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:07.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:07.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:07.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:07.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:07.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001910 con 0x7f63d8003400 2021-03-09T09:54:07.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:07.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00679f0 con 0x7f63d8008840 2021-03-09T09:54:07.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:07.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068710 con 0x7f6400146050 2021-03-09T09:54:07.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:07.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:07.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:54:07.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:07.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:07.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:07.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:07.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.792+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:07.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:07.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T09:54:07.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:07.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:07.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:07.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:54:07.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:54:07.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:07.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:07.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d840 con 0x7f6400146050 2021-03-09T09:54:07.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:07.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:07.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:54:07.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:54:07.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.793+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:07.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.794+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10837 cs=0 l=1 rev1=1 rx=0x7f63e006e270 tx=0x7f63f0006ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:07.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.794+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:54:07.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.794+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:07.807 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:07.796+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29747 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:07.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.796+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T09:54:07.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.796+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:07.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.797+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:07.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.797+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:07.809 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:07.809 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:07.810 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:07.810 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:07.810 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:07.810 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:07.810 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:07.811 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:07.811 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:07.811 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:07.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.797+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T09:54:07.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.797+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:07.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.797+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:54:07.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:07.799+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007fd90 con 0x7f6400146050 2021-03-09T09:54:08.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:08.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:08.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:08.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:08.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:08.592+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:09.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:09.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:09.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:09.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:10.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:10.103+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:10.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:10.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:10.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:10.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:11.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:11.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:11.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:11.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:12.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:12.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:12.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:12.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:12.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10837 cs=0 l=1 rev1=1 rx=0x7f63e006e270 tx=0x7f63f0006ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:12.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10837 cs=0 l=1 rev1=1 rx=0x7f63e006e270 tx=0x7f63f0006ca0).stop 2021-03-09T09:54:12.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:12.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:12.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:12.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:12.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:54:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:12.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:12.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:12.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:12.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:12.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005efd0 con 0x7f63d8003400 2021-03-09T09:54:12.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:12.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8008840 2021-03-09T09:54:12.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:12.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:54:12.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:12.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:12.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.797+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:12.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.798+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:12.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.798+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:54:12.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.798+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:12.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.798+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:12.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.798+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:12.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.798+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:12.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.799+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:12.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.799+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:12.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.799+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:12.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:12.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:12.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:12.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:12.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:12.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:54:12.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:54:12.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:12.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:12.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:54:12.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:12.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:12.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a900 con 0x7f63e0018390 2021-03-09T09:54:12.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:54:12.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.800+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:12.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.801+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:54:12.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.801+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:12.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.801+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10858 cs=0 l=1 rev1=1 rx=0x7f63e002e440 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:12.815 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:12.801+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29785 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:12.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f63e0018390 2021-03-09T09:54:12.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.802+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:12.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.802+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:12.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.802+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:12.818 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:12.818 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:12.819 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:12.820 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:12.820 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:12.820 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:12.821 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:12.821 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:12.822 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:12.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.806+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc008070 con 0x7f63e0018390 2021-03-09T09:54:12.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.806+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:12.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.806+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:54:12.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:12.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e0018390 2021-03-09T09:54:13.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:13.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:13.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:13.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:14.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:14.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:14.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:14.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:15.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:15.104+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:15.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:15.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:15.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:15.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:16.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:16.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:17.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:17.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:17.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:17.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:17.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10858 cs=0 l=1 rev1=1 rx=0x7f63e002e440 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:17.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10858 cs=0 l=1 rev1=1 rx=0x7f63e002e440 tx=0x7f63fc009a00).stop 2021-03-09T09:54:17.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:17.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:17.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:17.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:17.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:54:17.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:17.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:17.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:17.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:17.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:17.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:17.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:17.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00290f0 con 0x7f63d8003400 2021-03-09T09:54:17.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:17.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010cb0 con 0x7f63d8008840 2021-03-09T09:54:17.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:17.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f6400146050 2021-03-09T09:54:17.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:17.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:17.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:17.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.802+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:17.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.803+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:17.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.803+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:17.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.803+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:54:17.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.803+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:17.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.803+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:17.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:17.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:17.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:17.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:17.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:17.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:17.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:17.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:17.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:54:17.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:54:17.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:17.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:17.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017790 con 0x7f6400146050 2021-03-09T09:54:17.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:17.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:17.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d310 con 0x7f6400146050 2021-03-09T09:54:17.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:54:17.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.804+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:17.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.805+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:54:17.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.805+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:17.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.805+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10868 cs=0 l=1 rev1=1 rx=0x7f63f4061f60 tx=0x7f63f40701e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:17.820 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:17.806+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29823 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:17.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4016e00 con 0x7f6400146050 2021-03-09T09:54:17.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.808+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:17.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.808+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:17.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.808+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:17.823 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:17.823 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:17.823 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:17.823 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:17.824 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:17.824 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:17.824 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:17.824 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:17.825 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:17.825 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:17.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4016e00 con 0x7f6400146050 2021-03-09T09:54:17.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.812+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:17.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.812+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:54:17.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:17.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f6400146050 2021-03-09T09:54:18.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:18.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:18.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:18.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:18.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:18.593+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:19.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:19.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:19.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:19.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:20.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:20.105+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:20.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:20.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:20.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:20.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:21.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:21.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:21.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:21.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:22.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:22.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:22.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:22.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10868 cs=0 l=1 rev1=1 rx=0x7f63f4061f60 tx=0x7f63f40701e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:22.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10868 cs=0 l=1 rev1=1 rx=0x7f63f4061f60 tx=0x7f63f40701e0).stop 2021-03-09T09:54:22.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:22.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:22.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:22.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:22.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:54:22.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:22.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:22.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:22.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:22.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:22.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:22.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:22.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c0c0 con 0x7f63d8003400 2021-03-09T09:54:22.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:22.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fa80 con 0x7f63d8008840 2021-03-09T09:54:22.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:22.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.806+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:54:22.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:22.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:22.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:22.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:22.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:22.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:22.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:54:22.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:22.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:22.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:22.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:22.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:22.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:22.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:22.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:22.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:22.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:22.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:54:22.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:54:22.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:22.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:22.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T09:54:22.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:22.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:22.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d840 con 0x7f63e0018390 2021-03-09T09:54:22.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:54:22.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.807+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:22.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.808+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10878 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f0038e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:22.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.808+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:54:22.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.808+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:22.822 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:22.811+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29862 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:22.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.811+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T09:54:22.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.811+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:22.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.811+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:22.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.811+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:22.824 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:22.824 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:22.824 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:22.824 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:22.824 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:22.825 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:22.825 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:22.825 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:22.825 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:22.825 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:22.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.811+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0088ee0 con 0x7f63e0018390 2021-03-09T09:54:22.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.811+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:22.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.811+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:54:22.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:22.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038770 con 0x7f63e0018390 2021-03-09T09:54:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:23.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:23.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:24.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:24.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:24.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:24.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:25.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:25.106+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:25.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:25.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:25.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:25.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:26.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:26.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:26.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:26.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:27.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:27.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:54:27.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:27.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10878 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f0038e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:27.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10878 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f0038e80).stop 2021-03-09T09:54:27.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:27.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:27.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:27.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:27.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:54:27.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:27.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:27.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:27.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:27.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:27.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:27.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:27.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8003400 2021-03-09T09:54:27.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:27.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00290f0 con 0x7f63d8008840 2021-03-09T09:54:27.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:27.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400146050 2021-03-09T09:54:27.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:27.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:27.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:27.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:27.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:27.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:27.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:54:27.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:27.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.812+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:27.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:27.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:27.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:27.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:27.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:27.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:27.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:27.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:27.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:54:27.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:54:27.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:27.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:27.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d970 con 0x7f6400146050 2021-03-09T09:54:27.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:27.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:27.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:54:27.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:54:27.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.813+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:27.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.814+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10896 cs=0 l=1 rev1=1 rx=0x7f63e0023b10 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:27.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.814+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:54:27.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.814+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:27.827 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:27.816+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29900 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:27.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f6400146050 2021-03-09T09:54:27.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.817+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:27.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.817+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:27.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.817+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:27.828 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:27.829 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:27.829 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:27.829 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:27.829 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:27.829 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:27.830 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:27.830 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:27.830 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:27.830 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:27.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T09:54:27.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.817+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:27.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.818+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:54:27.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:27.818+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f6400146050 2021-03-09T09:54:28.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:28.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:28.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:28.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:28.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:28.595+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:29.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:29.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:29.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:29.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:30.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:30.107+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:30.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:30.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:30.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:30.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:31.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:31.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:31.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:31.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:32.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:32.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:32.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10896 cs=0 l=1 rev1=1 rx=0x7f63e0023b10 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:32.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10896 cs=0 l=1 rev1=1 rx=0x7f63e0023b10 tx=0x7f63fc04a6d0).stop 2021-03-09T09:54:32.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:32.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:32.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:32.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10896 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:32.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:54:32.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:32.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:32.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:32.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:32.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:32.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:32.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:32.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025cb0 con 0x7f63d8003400 2021-03-09T09:54:32.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:32.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013ee0 con 0x7f63d8008840 2021-03-09T09:54:32.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:32.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0018390 2021-03-09T09:54:32.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:32.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:32.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:32.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:32.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:54:32.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:32.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.817+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:32.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:32.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:32.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:32.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:32.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:32.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:32.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:32.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:32.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:32.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:32.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:54:32.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:54:32.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:32.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:32.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e2e0 con 0x7f63e0018390 2021-03-09T09:54:32.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:32.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:32.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d970 con 0x7f63e0018390 2021-03-09T09:54:32.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:54:32.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.818+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:32.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.819+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:54:32.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.819+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:32.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.819+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10907 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:32.831 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:32.820+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29939 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:32.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.821+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f63e0018390 2021-03-09T09:54:32.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.821+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:32.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.821+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:32.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.821+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:32.833 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:32.833 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:32.833 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:32.833 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:32.833 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:32.834 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:32.834 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:32.834 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:32.834 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:32.834 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:32.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.821+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065e60 con 0x7f63e0018390 2021-03-09T09:54:32.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.821+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:32.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.821+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:54:32.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:32.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012a30 con 0x7f63e0018390 2021-03-09T09:54:33.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:33.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:33.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:33.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:34.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:34.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:34.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:34.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:35.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:35.108+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:35.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:35.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:35.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:35.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:36.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:36.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:36.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:36.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:37.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:37.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:37.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:37.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:37.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10907 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:37.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10907 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63fc009a00).stop 2021-03-09T09:54:37.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:37.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:37.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:37.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:37.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:54:37.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079a50 con 0x7f63d8008840 2021-03-09T09:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:37.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00290f0 con 0x7f6400146050 2021-03-09T09:54:37.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.822+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:37.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:37.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:37.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:37.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:37.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:37.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:54:37.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:37.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.823+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:37.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:37.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015340 con 0x7f6400146050 2021-03-09T09:54:37.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:37.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:37.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017790 con 0x7f6400146050 2021-03-09T09:54:37.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:54:37.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:37.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10919 cs=0 l=1 rev1=1 rx=0x7f63f405bc00 tx=0x7f63f4061e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:37.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:54:37.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.824+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:37.835 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:37.826+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 29977 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:37.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.826+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:54:37.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.827+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:37.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.827+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:37.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.827+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:37.837 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:37.837 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:37.837 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:37.837 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:37.838 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:37.838 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:37.838 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:37.838 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:37.839 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:37.839 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:37.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.827+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:54:37.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.827+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:37.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.827+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:54:37.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:37.828+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f6400146050 2021-03-09T09:54:38.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:38.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:38.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:38.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:38.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:38.596+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:39.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:39.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:39.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:39.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:40.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:40.109+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:40.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:40.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:40.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:40.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:41.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:41.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:41.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:41.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:42.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:42.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:42.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:42.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:42.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10919 cs=0 l=1 rev1=1 rx=0x7f63f405bc00 tx=0x7f63f4061e10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:42.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10919 cs=0 l=1 rev1=1 rx=0x7f63f405bc00 tx=0x7f63f4061e10).stop 2021-03-09T09:54:42.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:42.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:42.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:42.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:42.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:54:42.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:42.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014540 con 0x7f63d8003400 2021-03-09T09:54:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067510 con 0x7f63d8008840 2021-03-09T09:54:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000eaf0 con 0x7f63e0018390 2021-03-09T09:54:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.827+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.828+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:42.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.828+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:54:42.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.828+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:42.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.828+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:42.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.828+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:42.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.828+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:42.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:42.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:42.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:42.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:42.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:42.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:42.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:42.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:42.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:54:42.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:54:42.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:42.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:42.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e350 con 0x7f63e0018390 2021-03-09T09:54:42.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:42.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:42.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T09:54:42.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:54:42.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.829+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:42.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.830+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10936 cs=0 l=1 rev1=1 rx=0x7f63f005d010 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:42.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.830+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:54:42.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.830+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:42.842 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:42.831+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30016 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:42.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e0018390 2021-03-09T09:54:42.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.832+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:42.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.832+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:42.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.832+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:42.844 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:42.844 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:42.846 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:42.846 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:42.846 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:42.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.835+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003b480 con 0x7f63e0018390 2021-03-09T09:54:42.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.835+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:42.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.836+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:54:42.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:42.837+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e0018390 2021-03-09T09:54:43.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:43.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:43.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:43.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:44.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:44.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:44.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:44.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:45.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:45.110+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:45.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:45.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:45.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:45.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:46.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:46.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:46.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:46.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:47.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:47.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:47.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.831+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:54:47.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.831+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:47.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10936 cs=0 l=1 rev1=1 rx=0x7f63f005d010 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:47.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10936 cs=0 l=1 rev1=1 rx=0x7f63f005d010 tx=0x7f63f0013b80).stop 2021-03-09T09:54:47.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:47.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:47.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:47.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:47.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:54:47.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:47.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:47.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:47.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c2c0 con 0x7f63d8003400 2021-03-09T09:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:47.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019670 con 0x7f63d8008840 2021-03-09T09:54:47.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:47.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:54:47.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:47.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:47.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:47.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.832+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.833+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.833+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.833+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.833+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.833+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.833+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.833+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:47.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:47.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:47.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:47.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:47.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:54:47.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:54:47.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:47.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:47.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0021740 con 0x7f6400146050 2021-03-09T09:54:47.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e350 con 0x7f6400146050 2021-03-09T09:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.834+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.836+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:54:47.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.836+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:47.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10952 cs=0 l=1 rev1=1 rx=0x7f63f007ea60 tx=0x7f63f004d110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:47.846 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:47.836+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30054 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:47.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038b90 con 0x7f6400146050 2021-03-09T09:54:47.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.837+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:47.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.837+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:47.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.837+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:47.849 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:47.849 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:47.849 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:47.849 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:47.849 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:47.850 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:47.850 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:47.850 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:47.850 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:47.850 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:47.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00541f0 con 0x7f6400146050 2021-03-09T09:54:47.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.843+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:47.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.843+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:54:47.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:47.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b90 con 0x7f6400146050 2021-03-09T09:54:48.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:48.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:48.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:48.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:48.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:48.597+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:49.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:49.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:49.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:49.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:50.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:50.111+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:50.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:50.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:50.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:50.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:51.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:51.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:51.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:51.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:52.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:52.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:52.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:52.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:52.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10952 cs=0 l=1 rev1=1 rx=0x7f63f007ea60 tx=0x7f63f004d110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:52.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10952 cs=0 l=1 rev1=1 rx=0x7f63f007ea60 tx=0x7f63f004d110).stop 2021-03-09T09:54:52.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:52.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:52.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:52.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:52.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:54:52.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:52.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:52.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:52.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:52.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:52.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:52.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:52.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015ea0 con 0x7f63d8003400 2021-03-09T09:54:52.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:52.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067510 con 0x7f63d8008840 2021-03-09T09:54:52.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:52.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0018390 2021-03-09T09:54:52.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:52.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:52.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:52.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.836+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:52.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.837+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:52.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.837+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:52.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.837+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:54:52.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.837+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:52.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.837+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:52.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.837+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:52.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.837+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:52.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.837+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:52.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:52.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:54:52.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:52.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:52.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010cb0 con 0x7f63e0018390 2021-03-09T09:54:52.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:52.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:52.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015340 con 0x7f63e0018390 2021-03-09T09:54:52.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:54:52.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:52.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10961 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:52.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:54:52.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.838+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:52.851 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:52.841+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30093 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:52.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:54:52.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.842+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:52.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.842+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:52.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.842+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:52.853 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:52.854 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:52.854 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:52.854 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:52.856 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:52.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:54:52.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.846+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:52.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.846+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:54:52.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:52.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404b780 con 0x7f63e0018390 2021-03-09T09:54:53.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:53.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:53.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:53.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:54.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:54.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:54.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:54.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:55.112+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:54:55.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:55.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:55.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:55.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:56.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:56.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:56.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:56.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:57.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:57.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:57.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:54:57.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:54:57.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10961 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:54:57.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10961 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4005110).stop 2021-03-09T09:54:57.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:57.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:54:57.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:54:57.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:54:57.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:54:57.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:57.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:54:57.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:57.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:54:57.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:54:57.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:54:57.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:57.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T09:54:57.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:57.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000eaf0 con 0x7f63d8008840 2021-03-09T09:54:57.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:54:57.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00290f0 con 0x7f6400146050 2021-03-09T09:54:57.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:54:57.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:54:57.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:54:57.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:57.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:54:57.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:54:57.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:54:57.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:54:57.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.841+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:54:57.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:54:57.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:54:57.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:54:57.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:54:57.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:57.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:57.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:54:57.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:54:57.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:54:57.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:54:57.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:57.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:57.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:54:57.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:54:57.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:54:57.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010cb0 con 0x7f6400146050 2021-03-09T09:54:57.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:54:57.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.842+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:57.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.843+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10973 cs=0 l=1 rev1=1 rx=0x7f63f4046fb0 tx=0x7f63f4034220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:54:57.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.843+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:54:57.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.843+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:54:57.855 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:54:57.845+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30131 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:54:57.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005c10 con 0x7f6400146050 2021-03-09T09:54:57.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.846+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:54:57.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.846+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:54:57.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.846+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:54:57.857 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:54:57.857 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:54:57.857 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:57.857 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:54:57.858 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:54:57.858 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:54:57.858 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:54:57.858 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:54:57.858 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:54:57.859 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:54:57.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f6400146050 2021-03-09T09:54:57.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.846+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:54:57.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.846+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:54:57.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:57.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005c10 con 0x7f6400146050 2021-03-09T09:54:58.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:58.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:58.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:58.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:54:58.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:58.599+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:54:59.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:59.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:54:59.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:54:59.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:00.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:00.113+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:00.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:00.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:00.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:00.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:01.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:01.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:01.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:01.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:02.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:02.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:02.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:02.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:02.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10973 cs=0 l=1 rev1=1 rx=0x7f63f4046fb0 tx=0x7f63f4034220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:02.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=10973 cs=0 l=1 rev1=1 rx=0x7f63f4046fb0 tx=0x7f63f4034220).stop 2021-03-09T09:55:02.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:02.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:02.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:02.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=10973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:02.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:55:02.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:02.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:02.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:02.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:02.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:02.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:02.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:02.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c2c0 con 0x7f63d8003400 2021-03-09T09:55:02.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:02.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bfc0 con 0x7f63d8008840 2021-03-09T09:55:02.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:02.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023590 con 0x7f63e0018390 2021-03-09T09:55:02.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:02.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:02.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:02.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:02.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:02.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.846+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:02.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.846+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:55:02.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.846+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:02.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.846+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:02.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:02.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:02.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:02.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:02.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:02.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:02.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:02.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:02.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:55:02.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:55:02.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:02.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:02.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:55:02.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:02.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:02.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.847+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:55:02.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.848+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:55:02.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.848+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:02.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.849+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10994 cs=0 l=1 rev1=1 rx=0x7f63f4046d80 tx=0x7f63f405bab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:02.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.849+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:55:02.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.849+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:02.862 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:02.849+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30170 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:02.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072470 con 0x7f63e0018390 2021-03-09T09:55:02.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.850+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:02.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.850+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:02.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.850+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:02.863 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:02.864 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:02.864 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:02.864 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:02.864 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:02.865 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:02.865 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:02.865 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:02.865 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:02.865 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f63e0018390 2021-03-09T09:55:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.851+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.851+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:55:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:02.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005c10 con 0x7f63e0018390 2021-03-09T09:55:03.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:03.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:03.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:03.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:04.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:04.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:04.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:04.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:05.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:05.113+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:05.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:05.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:05.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:05.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:06.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:06.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:06.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:06.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:07.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:07.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:07.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:07.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:07.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10994 cs=0 l=1 rev1=1 rx=0x7f63f4046d80 tx=0x7f63f405bab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:07.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=10994 cs=0 l=1 rev1=1 rx=0x7f63f4046d80 tx=0x7f63f405bab0).stop 2021-03-09T09:55:07.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:07.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:07.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:07.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=10994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:07.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:55:07.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:07.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:07.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:07.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:07.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:07.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:07.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:07.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025340 con 0x7f63d8003400 2021-03-09T09:55:07.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:07.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015ea0 con 0x7f63d8008840 2021-03-09T09:55:07.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:07.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:55:07.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:07.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:07.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:07.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:07.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:07.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.852+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:07.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.852+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:55:07.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.852+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:07.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.852+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:07.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.852+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:07.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.852+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:07.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.852+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:07.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:07.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:07.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:07.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:07.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:07.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:55:07.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:55:07.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:07.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:07.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e350 con 0x7f6400146050 2021-03-09T09:55:07.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:07.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:07.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e2e0 con 0x7f6400146050 2021-03-09T09:55:07.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:55:07.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:07.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11008 cs=0 l=1 rev1=1 rx=0x7f63e0001e80 tx=0x7f63fc04bd20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:07.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:55:07.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.853+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:07.867 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:07.854+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30208 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:07.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f6400146050 2021-03-09T09:55:07.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.856+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:07.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.856+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:07.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.856+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:07.869 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:07.869 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:07.869 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:07.869 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:07.869 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:07.870 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:07.870 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:07.870 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:07.870 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:07.870 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:07.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.857+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fab0 con 0x7f6400146050 2021-03-09T09:55:07.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.857+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:07.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.858+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:55:07.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:07.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T09:55:08.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:08.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:08.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:08.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:08.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:08.599+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:09.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:09.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:09.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:10.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:10.114+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:10.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:10.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:10.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:10.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:11.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:11.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:11.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:11.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:12.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:12.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:12.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.854+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:12.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:12.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11008 cs=0 l=1 rev1=1 rx=0x7f63e0001e80 tx=0x7f63fc04bd20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:12.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11008 cs=0 l=1 rev1=1 rx=0x7f63e0001e80 tx=0x7f63fc04bd20).stop 2021-03-09T09:55:12.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:12.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:12.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:12.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:12.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:55:12.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:12.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:12.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:12.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:12.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:12.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:12.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:12.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c2c0 con 0x7f63d8003400 2021-03-09T09:55:12.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:12.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067510 con 0x7f63d8008840 2021-03-09T09:55:12.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:12.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026010 con 0x7f63e0018390 2021-03-09T09:55:12.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:12.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:12.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:12.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:12.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.855+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:12.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.856+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:12.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.856+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:55:12.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.856+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:12.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.856+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:12.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.856+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:12.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.856+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:12.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.856+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:12.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:12.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:12.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:12.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:12.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:12.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:55:12.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:55:12.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:12.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:12.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:55:12.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:12.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:12.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e350 con 0x7f63e0018390 2021-03-09T09:55:12.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:55:12.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:12.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11020 cs=0 l=1 rev1=1 rx=0x7f63e0010920 tx=0x7f63fc06fb30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:12.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:55:12.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.858+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:12.872 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:12.859+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30247 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:12.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0018390 2021-03-09T09:55:12.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.860+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:12.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.860+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:12.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.860+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:12.875 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:12.875 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:12.875 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:12.875 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:12.876 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:12.876 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:12.876 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:12.877 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:12.877 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:12.877 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:12.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63e0018390 2021-03-09T09:55:12.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.861+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:12.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.861+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:55:12.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:12.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0018390 2021-03-09T09:55:13.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:13.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:13.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:13.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:14.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:14.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:14.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:14.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:15.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:15.115+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:15.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:15.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:15.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:15.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:16.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:16.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:16.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:16.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:17.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:17.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:17.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:17.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:17.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.859+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11020 cs=0 l=1 rev1=1 rx=0x7f63e0010920 tx=0x7f63fc06fb30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:17.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.859+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11020 cs=0 l=1 rev1=1 rx=0x7f63e0010920 tx=0x7f63fc06fb30).stop 2021-03-09T09:55:17.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:17.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:17.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:17.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:17.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:55:17.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:17.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:17.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:17.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:17.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:17.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:17.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:17.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002baf0 con 0x7f63d8003400 2021-03-09T09:55:17.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:17.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00237d0 con 0x7f63d8008840 2021-03-09T09:55:17.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:17.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:55:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:17.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:17.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:17.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.860+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.861+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.861+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:55:17.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.861+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:17.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.861+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:17.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:17.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:17.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:17.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:17.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:17.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:17.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:55:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:55:17.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f120 con 0x7f6400146050 2021-03-09T09:55:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:17.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:17.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0021740 con 0x7f6400146050 2021-03-09T09:55:17.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:55:17.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:17.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:55:17.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11033 cs=0 l=1 rev1=1 rx=0x7f63e004e0a0 tx=0x7f63f0055f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:17.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.863+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:17.877 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:17.864+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30285 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:17.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f6400146050 2021-03-09T09:55:17.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.864+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:17.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.864+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:17.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.864+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:17.879 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:17.879 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:17.879 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:17.879 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:17.880 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:17.880 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:17.880 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:17.880 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:17.880 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:17.881 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:17.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f000dee0 con 0x7f6400146050 2021-03-09T09:55:17.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.865+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:17.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.865+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:55:17.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:17.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:55:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:18.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:18.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:18.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:18.600+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:19.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:19.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:19.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:19.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:20.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:20.116+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:20.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:20.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:20.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:20.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:21.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:21.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:21.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:21.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:22.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:22.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:22.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:55:22.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:22.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11033 cs=0 l=1 rev1=1 rx=0x7f63e004e0a0 tx=0x7f63f0055f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:22.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11033 cs=0 l=1 rev1=1 rx=0x7f63e004e0a0 tx=0x7f63f0055f90).stop 2021-03-09T09:55:22.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:22.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:22.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:22.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:55:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:22.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:22.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025340 con 0x7f63d8003400 2021-03-09T09:55:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015ea0 con 0x7f63d8008840 2021-03-09T09:55:22.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c2c0 con 0x7f63e0018390 2021-03-09T09:55:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.864+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:22.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:22.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:22.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:22.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:55:22.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:22.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:22.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:22.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:22.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:22.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:22.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:55:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:55:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:22.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:22.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d710 con 0x7f63e0018390 2021-03-09T09:55:22.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:22.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:22.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.865+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:55:22.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.866+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:55:22.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.866+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:22.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.866+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002e630 secure :-1 s=READY pgs=11043 cs=0 l=1 rev1=1 rx=0x7f63fc075fb0 tx=0x7f63fc0039e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:22.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.866+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:55:22.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.866+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:22.881 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:22.868+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30324 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:22.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc05deb0 con 0x7f63e0018390 2021-03-09T09:55:22.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.870+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:22.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.870+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:22.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.870+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:22.883 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:22.883 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:22.884 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:22.884 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:22.884 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:22.884 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:22.884 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:22.885 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:22.885 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:22.885 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:22.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06fee0 con 0x7f63e0018390 2021-03-09T09:55:22.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.870+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:22.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.870+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:55:22.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:22.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63e0018390 2021-03-09T09:55:23.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:23.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:23.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:23.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:24.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:24.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:24.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:24.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:25.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:25.117+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:25.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:25.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:25.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:25.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:26.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:26.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:26.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:26.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:27.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:27.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:27.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e002e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:27.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e002e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:27.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002e630 secure :-1 s=READY pgs=11043 cs=0 l=1 rev1=1 rx=0x7f63fc075fb0 tx=0x7f63fc0039e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:27.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002e630 secure :-1 s=READY pgs=11043 cs=0 l=1 rev1=1 rx=0x7f63fc075fb0 tx=0x7f63fc0039e0).stop 2021-03-09T09:55:27.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:27.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:27.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e002e630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:27.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002e630 unknown :-1 s=CLOSED pgs=11043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:27.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:55:27.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:27.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:27.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:27.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:27.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:27.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:27.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:27.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:55:27.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:27.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T09:55:27.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:27.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001dab0 con 0x7f6400146050 2021-03-09T09:55:27.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:27.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:27.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.869+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:27.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.870+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:27.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.870+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:27.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.870+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:27.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.870+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:55:27.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.870+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:27.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.870+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:27.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.871+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:27.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.871+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:27.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.871+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:27.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:27.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:27.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:27.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:27.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:27.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:55:27.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:55:27.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:27.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:27.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:55:27.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:27.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:27.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f120 con 0x7f6400146050 2021-03-09T09:55:27.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:55:27.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.872+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:27.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11061 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:27.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.873+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:55:27.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.873+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:27.884 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:27.873+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30362 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:27.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:55:27.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.874+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.874+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.874+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:27.886 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:27.887 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:27.887 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:27.887 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:27.887 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:27.887 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:27.888 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:27.888 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:27.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.875+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:55:27.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.875+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:27.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.875+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:55:27.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:27.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T09:55:28.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:28.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:28.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:28.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:28.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:28.602+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:29.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:29.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:29.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:29.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:30.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:30.118+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:30.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:30.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:30.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:30.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:31.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:31.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:31.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:31.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:32.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:32.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:32.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:32.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:32.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11061 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:32.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11061 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63f000df50).stop 2021-03-09T09:55:32.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:32.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:32.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:32.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11061 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:32.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:55:32.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:32.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:32.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:32.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:32.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:32.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:32.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:32.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T09:55:32.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:32.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:55:32.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:32.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00237d0 con 0x7f63e0018390 2021-03-09T09:55:32.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:32.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:32.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:32.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:32.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:32.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:32.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:55:32.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:32.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.874+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:32.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:32.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:32.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:32.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:32.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:32.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:32.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:32.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:32.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:55:32.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:55:32.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:32.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:32.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e350 con 0x7f63e0018390 2021-03-09T09:55:32.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:32.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:32.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:55:32.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.876+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:55:32.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.876+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:32.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.876+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11068 cs=0 l=1 rev1=1 rx=0x7f63f0013b50 tx=0x7f63f0042f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:32.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.876+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:55:32.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.876+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:32.890 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:32.878+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30401 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:32.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028c70 con 0x7f63e0018390 2021-03-09T09:55:32.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.879+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:32.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.879+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:32.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.879+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:32.892 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:32.892 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:32.893 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:32.893 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:32.893 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:32.893 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:32.893 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:32.894 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:32.894 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:32.894 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:32.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028c70 con 0x7f63e0018390 2021-03-09T09:55:32.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.880+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:32.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.880+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:55:32.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:32.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005ed20 con 0x7f63e0018390 2021-03-09T09:55:33.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:33.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:33.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:33.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:34.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:34.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:34.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:34.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:35.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:35.119+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:35.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:35.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:35.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:35.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:36.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:36.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:36.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:36.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:37.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:37.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:37.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11068 cs=0 l=1 rev1=1 rx=0x7f63f0013b50 tx=0x7f63f0042f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:37.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11068 cs=0 l=1 rev1=1 rx=0x7f63f0013b50 tx=0x7f63f0042f30).stop 2021-03-09T09:55:37.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:37.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:37.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:37.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:37.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:55:37.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:37.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:37.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:37.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:37.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:37.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:37.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:37.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025340 con 0x7f63d8003400 2021-03-09T09:55:37.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:37.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63d8008840 2021-03-09T09:55:37.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:37.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f6400146050 2021-03-09T09:55:37.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:37.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:37.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:37.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:37.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:37.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.880+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:37.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.880+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:55:37.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.880+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:37.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.880+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:37.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:37.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:37.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:37.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:37.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:37.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:37.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:37.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:37.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:55:37.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:55:37.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:37.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:37.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:55:37.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:37.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:37.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e350 con 0x7f6400146050 2021-03-09T09:55:37.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:55:37.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.881+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:37.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.882+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11087 cs=0 l=1 rev1=1 rx=0x7f63f007f000 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:37.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.882+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:55:37.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.882+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:37.895 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:37.883+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30440 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:37.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005dc70 con 0x7f6400146050 2021-03-09T09:55:37.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.884+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:37.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.884+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:37.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.884+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:37.897 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:37.897 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:37.897 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:37.897 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:37.898 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:37.898 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:37.898 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:37.898 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:37.899 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:37.899 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:37.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005dc70 con 0x7f6400146050 2021-03-09T09:55:37.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.887+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:37.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.887+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:55:37.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:37.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f6400146050 2021-03-09T09:55:38.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:38.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:38.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:38.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:38.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:38.603+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:39.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:39.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:39.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:39.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:40.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:40.120+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:40.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:40.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:40.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:40.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:41.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:41.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:41.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:41.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:42.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:42.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:55:42.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:42.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11087 cs=0 l=1 rev1=1 rx=0x7f63f007f000 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:42.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11087 cs=0 l=1 rev1=1 rx=0x7f63f007f000 tx=0x7f63f004dd90).stop 2021-03-09T09:55:42.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:42.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:42.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:42.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:42.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:55:42.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:42.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:42.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:42.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:42.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:42.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:42.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:42.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f63d8003400 2021-03-09T09:55:42.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:42.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c8a0 con 0x7f63d8008840 2021-03-09T09:55:42.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:42.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:55:42.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:42.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:42.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:42.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:42.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.884+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:42.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:42.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:55:42.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:42.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:42.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:42.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:42.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:42.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:42.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:42.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:55:42.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.885+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:55:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015e50 con 0x7f63e0018390 2021-03-09T09:55:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:42.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:55:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:55:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:55:42.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11096 cs=0 l=1 rev1=1 rx=0x7f63f006d460 tx=0x7f63f004d1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:42.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.886+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:42.900 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:42.889+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30479 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:42.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00884c0 con 0x7f63e0018390 2021-03-09T09:55:42.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.889+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:42.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.889+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:42.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.889+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:42.902 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:42.902 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:42.902 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:42.902 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:42.903 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:42.903 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:42.903 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:42.903 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:42.904 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:42.904 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:42.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63e0018390 2021-03-09T09:55:42.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.889+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:42.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.889+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:55:42.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:42.890+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042bc0 con 0x7f63e0018390 2021-03-09T09:55:43.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:43.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:43.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:43.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:44.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:44.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:44.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:45.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:45.121+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:45.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:45.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:45.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:45.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:46.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:46.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:46.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:46.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:47.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:47.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:47.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:47.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:47.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11096 cs=0 l=1 rev1=1 rx=0x7f63f006d460 tx=0x7f63f004d1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:47.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11096 cs=0 l=1 rev1=1 rx=0x7f63f006d460 tx=0x7f63f004d1f0).stop 2021-03-09T09:55:47.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:47.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:47.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:47.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:47.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:55:47.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:47.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:47.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:47.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:47.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:47.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:47.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:47.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001b10 con 0x7f63d8003400 2021-03-09T09:55:47.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:47.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c2c0 con 0x7f63d8008840 2021-03-09T09:55:47.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:47.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025340 con 0x7f6400146050 2021-03-09T09:55:47.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.889+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:47.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:47.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:47.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:47.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:55:47.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:47.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:47.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:47.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:47.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:47.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:47.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:47.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:47.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:47.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:47.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:47.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:47.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:55:47.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:55:47.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:47.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:47.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:55:47.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:47.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:47.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015e50 con 0x7f6400146050 2021-03-09T09:55:47.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:55:47.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.890+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:47.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.891+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11109 cs=0 l=1 rev1=1 rx=0x7f63e001d8a0 tx=0x7f63f0028ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:47.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.891+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:55:47.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.891+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:47.904 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:47.894+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30518 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:47.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T09:55:47.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.895+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:47.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.895+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:47.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.895+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:47.906 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:47.906 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:47.906 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:47.906 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:47.906 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:47.907 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:47.907 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:47.907 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:47.907 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:47.908 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:47.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00888f0 con 0x7f6400146050 2021-03-09T09:55:47.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.895+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:47.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.895+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:55:47.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:47.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042bc0 con 0x7f6400146050 2021-03-09T09:55:48.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:48.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:48.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:48.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:48.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:48.604+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:49.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:49.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:49.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:49.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:50.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:50.122+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:50.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:50.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:50.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:50.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:51.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:51.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:51.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:51.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:52.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:52.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:52.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:52.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:52.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11109 cs=0 l=1 rev1=1 rx=0x7f63e001d8a0 tx=0x7f63f0028ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:52.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11109 cs=0 l=1 rev1=1 rx=0x7f63e001d8a0 tx=0x7f63f0028ea0).stop 2021-03-09T09:55:52.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:52.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:52.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:52.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:52.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:55:52.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:52.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:52.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:52.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:52.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:52.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:52.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:52.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a750 con 0x7f63d8003400 2021-03-09T09:55:52.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:52.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028bc0 con 0x7f63d8008840 2021-03-09T09:55:52.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:52.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001dab0 con 0x7f63e0018390 2021-03-09T09:55:52.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:52.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:52.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:52.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:52.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:52.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.894+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:52.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:55:52.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:52.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:52.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:52.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:52.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:52.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:52.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:52.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:52.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:52.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:52.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:55:52.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:55:52.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:52.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:52.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce40 con 0x7f63e0018390 2021-03-09T09:55:52.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:52.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:52.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:55:52.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:55:52.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.895+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:52.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.896+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:55:52.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.896+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11124 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f0073fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:52.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.896+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:52.910 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:52.898+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30557 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:52.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63e0018390 2021-03-09T09:55:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.899+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.899+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.899+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:52.911 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:52.912 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:52.913 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:52.913 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63e0018390 2021-03-09T09:55:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.899+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.899+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:55:52.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:52.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00112f0 con 0x7f63e0018390 2021-03-09T09:55:53.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:53.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:53.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:53.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:54.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:54.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:54.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:54.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:55.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:55.123+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:55:55.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:55.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:55.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:55.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:56.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:56.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:56.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:57.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:57.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:57.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:55:57.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:55:57.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11124 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f0073fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:55:57.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11124 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f0073fd0).stop 2021-03-09T09:55:57.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:57.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:55:57.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:55:57.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:55:57.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:55:57.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:57.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:55:57.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:57.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:55:57.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:55:57.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:55:57.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:57.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e390 con 0x7f63d8003400 2021-03-09T09:55:57.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:57.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013300 con 0x7f63d8008840 2021-03-09T09:55:57.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:55:57.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:55:57.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:55:57.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:55:57.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.899+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:57.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:55:57.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.900+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:55:57.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.900+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:55:57.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.900+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:55:57.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.900+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:55:57.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.900+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:55:57.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.900+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:55:57.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.900+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:55:57.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.900+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:55:57.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:55:57.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:57.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:57.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:55:57.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:55:57.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:55:57.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:55:57.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:57.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:57.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:55:57.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:55:57.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:55:57.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d710 con 0x7f6400146050 2021-03-09T09:55:57.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:55:57.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:57.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11138 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:55:57.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:55:57.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.901+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:55:57.915 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:55:57.903+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30596 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:55:57.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058410 con 0x7f6400146050 2021-03-09T09:55:57.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.904+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:55:57.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.905+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:55:57.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.905+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:55:57.916 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:55:57.917 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:55:57.917 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:57.917 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:55:57.917 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:55:57.918 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:55:57.918 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:55:57.918 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:55:57.918 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:55:57.918 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:55:57.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058410 con 0x7f6400146050 2021-03-09T09:55:57.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.907+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:55:57.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.907+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:55:57.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:57.908+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:55:58.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:58.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:58.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:58.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:55:58.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:58.605+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:55:59.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:59.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:55:59.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:55:59.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:00.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:00.124+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:00.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:00.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:00.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:00.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:01.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:01.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:01.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:01.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:02.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:02.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:02.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:02.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:02.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11138 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:02.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11138 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a740).stop 2021-03-09T09:56:02.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:02.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:02.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:02.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:02.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:56:02.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:02.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:02.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:02.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:02.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:02.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:02.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:02.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c8a0 con 0x7f63d8003400 2021-03-09T09:56:02.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:02.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001b10 con 0x7f63d8008840 2021-03-09T09:56:02.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:02.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028bc0 con 0x7f63e0018390 2021-03-09T09:56:02.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:02.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:02.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:02.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:02.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.904+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:02.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.905+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:02.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.905+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:56:02.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.905+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:02.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.905+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:02.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.905+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:02.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.905+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:02.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.905+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:02.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:02.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:02.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:02.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:02.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:02.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:56:02.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:56:02.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:02.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:02.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001910 con 0x7f63e0018390 2021-03-09T09:56:02.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:02.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:02.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:56:02.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:56:02.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.906+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:02.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.907+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024ce0 secure :-1 s=READY pgs=11153 cs=0 l=1 rev1=1 rx=0x7f63f40341f0 tx=0x7f63f4005260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:02.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.907+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:56:02.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.907+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:02.920 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:02.909+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30635 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:02.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.910+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:56:02.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.910+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:02.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.910+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:02.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.910+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:02.921 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:02.922 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:02.922 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:02.922 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:02.922 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:02.922 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:02.923 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:02.923 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:02.923 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:02.923 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:02.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:56:02.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.916+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:02.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f63e0018390 2021-03-09T09:56:02.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:02.916+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:56:03.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:03.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:03.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:03.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:04.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:04.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:04.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:04.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:05.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:05.125+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:05.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:05.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:05.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:05.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:06.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:06.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:06.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:06.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:07.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:07.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:07.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0024ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0024ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024ce0 secure :-1 s=READY pgs=11153 cs=0 l=1 rev1=1 rx=0x7f63f40341f0 tx=0x7f63f4005260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024ce0 secure :-1 s=READY pgs=11153 cs=0 l=1 rev1=1 rx=0x7f63f40341f0 tx=0x7f63f4005260).stop 2021-03-09T09:56:07.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0024ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024ce0 unknown :-1 s=CLOSED pgs=11153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:56:07.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:07.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:07.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:07.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:07.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:56:07.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:07.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e390 con 0x7f63d8008840 2021-03-09T09:56:07.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:07.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013300 con 0x7f6400146050 2021-03-09T09:56:07.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.909+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:07.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.910+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:07.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.910+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:07.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.910+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:07.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.910+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.911+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.911+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:56:07.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.911+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.911+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.911+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.911+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:07.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.911+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:07.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:07.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:07.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:56:07.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:56:07.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:07.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e350 con 0x7f6400146050 2021-03-09T09:56:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:56:07.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:56:07.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.912+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:07.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.913+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:56:07.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.913+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:07.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.913+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11173 cs=0 l=1 rev1=1 rx=0x7f63e0012f20 tx=0x7f63fc065e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:07.926 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:07.914+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30674 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:07.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:56:07.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.914+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:07.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.914+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:07.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.914+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:07.927 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:07.928 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:07.928 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:07.928 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:07.928 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:07.928 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:07.929 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:07.929 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:07.929 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:07.929 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:07.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc006480 con 0x7f6400146050 2021-03-09T09:56:07.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.921+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:07.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:56:07.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:07.921+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:56:08.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:08.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:08.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:08.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:08.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:08.607+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:09.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:09.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:09.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:09.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:10.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:10.126+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:10.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:10.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:10.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:10.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:11.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:11.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:11.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:12.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:12.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:12.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:12.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:12.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11173 cs=0 l=1 rev1=1 rx=0x7f63e0012f20 tx=0x7f63fc065e40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:12.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11173 cs=0 l=1 rev1=1 rx=0x7f63e0012f20 tx=0x7f63fc065e40).stop 2021-03-09T09:56:12.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:12.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:12.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:12.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:12.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:56:12.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:12.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:12.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:12.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:12.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:12.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:12.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:12.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068f50 con 0x7f63d8003400 2021-03-09T09:56:12.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:12.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ee80 con 0x7f63d8008840 2021-03-09T09:56:12.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:12.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025340 con 0x7f63e0018390 2021-03-09T09:56:12.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:12.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:12.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:12.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:12.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:12.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.916+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:12.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.916+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:56:12.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.916+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:12.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.916+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:12.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.917+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:12.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.917+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:12.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.917+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:12.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:12.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:12.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:12.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:12.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:12.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:56:12.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:56:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c420 con 0x7f63e0018390 2021-03-09T09:56:12.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:12.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:12.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce40 con 0x7f63e0018390 2021-03-09T09:56:12.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:56:12.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:12.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11188 cs=0 l=1 rev1=1 rx=0x7f63f0074040 tx=0x7f63f005ea20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:12.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:56:12.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.918+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:12.930 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:12.919+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30713 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T09:56:12.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.919+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:12.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.919+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:12.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.919+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:12.932 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:12.932 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:12.932 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:12.933 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:12.933 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:12.933 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:12.933 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:12.934 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:12.934 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:12.934 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:12.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T09:56:12.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.920+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:12.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.920+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:56:12.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:12.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63e0018390 2021-03-09T09:56:13.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:13.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:13.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:13.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:14.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:14.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:14.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:14.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:15.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:15.127+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:15.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:15.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:15.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:15.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:16.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:16.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:16.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:16.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:17.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:17.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:17.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:56:17.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:17.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11188 cs=0 l=1 rev1=1 rx=0x7f63f0074040 tx=0x7f63f005ea20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:17.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11188 cs=0 l=1 rev1=1 rx=0x7f63f0074040 tx=0x7f63f005ea20).stop 2021-03-09T09:56:17.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:17.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:17.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:17.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:17.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:56:17.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:17.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:17.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:17.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:17.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:17.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:17.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:17.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ffe0 con 0x7f63d8003400 2021-03-09T09:56:17.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:17.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63d8008840 2021-03-09T09:56:17.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:17.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:56:17.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.919+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:17.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:17.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:17.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:17.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:17.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:17.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:56:17.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:17.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:17.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:17.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:17.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.920+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:17.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:17.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:17.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:17.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:17.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:17.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:56:17.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:56:17.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:17.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:17.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:56:17.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:17.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:17.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e350 con 0x7f6400146050 2021-03-09T09:56:17.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:56:17.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:17.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:56:17.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:17.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11191 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc0333f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:17.936 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:17.923+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30752 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:17.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02df10 con 0x7f6400146050 2021-03-09T09:56:17.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.924+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:17.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.924+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:17.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.924+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:17.938 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:17.938 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:17.938 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:17.939 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:17.939 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:17.939 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:17.939 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:17.939 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:17.940 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:17.940 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:17.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02df10 con 0x7f6400146050 2021-03-09T09:56:17.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.928+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:17.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.928+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:56:17.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:17.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:56:18.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:18.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:18.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:18.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:18.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:18.608+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:19.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:19.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:19.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:19.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:20.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:20.128+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:20.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:20.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:20.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:20.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:21.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:21.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:21.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:21.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:22.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:22.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:22.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:22.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:22.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11191 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc0333f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:22.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11191 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc0333f0).stop 2021-03-09T09:56:22.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:22.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:22.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:22.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:22.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:56:22.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:22.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:22.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:22.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:22.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:22.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:22.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:22.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc60 con 0x7f63d8003400 2021-03-09T09:56:22.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:22.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068f50 con 0x7f63d8008840 2021-03-09T09:56:22.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:22.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067710 con 0x7f63e0018390 2021-03-09T09:56:22.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:22.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:22.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:22.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:22.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:22.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:22.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:56:22.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:22.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.924+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:22.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:22.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:22.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:22.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:22.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:22.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:22.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:22.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:22.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:56:22.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:56:22.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:22.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:22.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005eda0 con 0x7f63e0018390 2021-03-09T09:56:22.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:22.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:22.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001910 con 0x7f63e0018390 2021-03-09T09:56:22.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:56:22.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.925+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:22.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.926+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11208 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4003b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:22.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.926+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:56:22.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.926+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:22.940 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:22.928+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30791 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:22.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T09:56:22.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.929+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:22.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.929+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:22.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.929+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:22.942 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:22.942 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:22.943 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:22.943 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:22.943 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:22.943 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:22.943 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:22.944 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:22.944 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:22.944 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:22.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005480 con 0x7f63e0018390 2021-03-09T09:56:22.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.932+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:22.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.932+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:56:22.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:22.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T09:56:23.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:23.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:23.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:24.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:24.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:24.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:24.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:25.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:25.129+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:25.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:25.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:25.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:25.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:26.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:26.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:26.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:26.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:27.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:27.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:27.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:27.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:27.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11208 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4003b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:27.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11208 cs=0 l=1 rev1=1 rx=0x7f63f4018010 tx=0x7f63f4003b40).stop 2021-03-09T09:56:27.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:27.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:27.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:27.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:27.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:56:27.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:27.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:27.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:27.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:27.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:27.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:27.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:27.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013300 con 0x7f63d8003400 2021-03-09T09:56:27.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:27.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:56:27.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f6400146050 2021-03-09T09:56:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:27.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:27.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:27.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:27.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:27.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:56:27.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:27.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.929+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:27.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:27.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:27.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:27.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:27.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:27.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:27.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:27.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:27.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:56:27.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:56:27.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:27.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:27.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f6400146050 2021-03-09T09:56:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005eda0 con 0x7f6400146050 2021-03-09T09:56:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:56:27.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.930+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:27.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.931+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11219 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f401af70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:27.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.931+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:56:27.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.931+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:27.945 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:27.934+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30830 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:27.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405be80 con 0x7f6400146050 2021-03-09T09:56:27.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.934+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:27.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.934+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:27.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.934+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:27.946 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:27.947 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:27.947 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:27.947 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:27.947 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:27.948 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:27.948 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:27.948 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:27.948 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:27.948 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:27.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022870 con 0x7f6400146050 2021-03-09T09:56:27.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.934+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:27.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.934+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:56:27.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:27.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40229e0 con 0x7f6400146050 2021-03-09T09:56:28.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:28.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:28.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:28.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:28.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:28.610+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:29.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:29.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:29.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:29.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:30.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:30.130+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:30.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:30.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:30.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:30.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:31.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:31.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:31.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:31.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:32.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:32.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:32.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:32.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:32.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11219 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f401af70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:32.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11219 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f401af70).stop 2021-03-09T09:56:32.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:32.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:32.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:32.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:32.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:56:32.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00198f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:32.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:32.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:32.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:32.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:32.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:32.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:32.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:56:32.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:32.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63d8008840 2021-03-09T09:56:32.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:32.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:56:32.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:32.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:32.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:32.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:32.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:32.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00198f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:32.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:56:32.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:32.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:32.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:32.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:32.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.935+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:32.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:32.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:32.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:32.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:32.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:32.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:56:32.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:56:32.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:32.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:32.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T09:56:32.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:32.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:32.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f63e0018390 2021-03-09T09:56:32.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:56:32.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:32.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00198f0 secure :-1 s=READY pgs=11233 cs=0 l=1 rev1=1 rx=0x7f63f4046e10 tx=0x7f63f405bb90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:32.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:56:32.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.936+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:32.951 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:32.939+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30869 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:32.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.941+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40483c0 con 0x7f63e0018390 2021-03-09T09:56:32.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.941+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:32.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.941+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:32.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.941+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:32.953 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:32.954 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:32.954 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:32.954 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:32.954 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:32.955 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:32.955 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:32.955 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:32.955 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:32.955 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:32.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40483c0 con 0x7f63e0018390 2021-03-09T09:56:32.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.943+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:32.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63e0018390 2021-03-09T09:56:32.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:32.944+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:56:33.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:33.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:33.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:33.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:34.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:34.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:34.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:34.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:35.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:35.131+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:35.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:35.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:35.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:35.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:36.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:36.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:36.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:36.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:37.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:37.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:37.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00198f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:37.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00198f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:37.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00198f0 secure :-1 s=READY pgs=11233 cs=0 l=1 rev1=1 rx=0x7f63f4046e10 tx=0x7f63f405bb90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:37.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00198f0 secure :-1 s=READY pgs=11233 cs=0 l=1 rev1=1 rx=0x7f63f4046e10 tx=0x7f63f405bb90).stop 2021-03-09T09:56:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00198f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00198f0 unknown :-1 s=CLOSED pgs=11233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:56:37.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:37.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:37.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:37.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:37.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:37.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:37.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:37.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00663a0 con 0x7f63d8003400 2021-03-09T09:56:37.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:37.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016db0 con 0x7f63d8008840 2021-03-09T09:56:37.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:37.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ab0 con 0x7f6400146050 2021-03-09T09:56:37.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:37.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:37.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:37.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:37.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.940+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:37.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:37.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:56:37.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:37.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:37.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:37.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:37.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:37.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:37.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:37.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:37.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:37.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:37.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:56:37.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.942+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:56:37.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:37.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:37.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017c90 con 0x7f6400146050 2021-03-09T09:56:37.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:37.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:37.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c420 con 0x7f6400146050 2021-03-09T09:56:37.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:56:37.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:37.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=11255 cs=0 l=1 rev1=1 rx=0x7f63f0029000 tx=0x7f63f0006ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:37.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:56:37.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.943+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:37.955 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:37.944+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30908 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:37.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f6400146050 2021-03-09T09:56:37.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.945+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:37.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.945+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:37.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.945+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:37.957 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:37.957 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:37.957 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:37.958 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:37.958 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:37.958 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:37.958 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:37.959 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:37.959 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:37.959 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:37.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f6400146050 2021-03-09T09:56:37.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.950+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:37.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.950+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:56:37.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:37.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f6400146050 2021-03-09T09:56:38.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:38.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:38.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:38.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:38.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:38.611+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:39.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:39.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:39.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:39.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:40.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:40.132+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:40.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:40.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:40.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:40.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:41.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:41.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:41.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:41.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:42.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:42.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:42.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:42.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=11255 cs=0 l=1 rev1=1 rx=0x7f63f0029000 tx=0x7f63f0006ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:42.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 secure :-1 s=READY pgs=11255 cs=0 l=1 rev1=1 rx=0x7f63f0029000 tx=0x7f63f0006ca0).stop 2021-03-09T09:56:42.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:42.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:42.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:42.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064b10 unknown :-1 s=CLOSED pgs=11255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:42.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:56:42.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:42.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:42.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e007b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022d80 con 0x7f63d8003400 2021-03-09T09:56:42.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:42.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:56:42.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:42.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d940 con 0x7f63e0018390 2021-03-09T09:56:42.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:42.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:42.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:42.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:42.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.945+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:42.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.946+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e007b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:42.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.946+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:56:42.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.946+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:42.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.946+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:42.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.947+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:42.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.947+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:42.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.947+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:42.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:42.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:42.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:42.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:42.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:42.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:56:42.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:56:42.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:42.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:42.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00658a0 con 0x7f63e0018390 2021-03-09T09:56:42.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:42.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:42.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:56:42.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:56:42.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:42.961 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30948 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:42.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.949+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e007b9f0 secure :-1 s=READY pgs=11269 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:42.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.950+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:56:42.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.950+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:42.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T09:56:42.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.950+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:42.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.950+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:42.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.950+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:42.963 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:42.964 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:42.964 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:42.964 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:42.964 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:42.964 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:42.965 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:42.965 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:42.965 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:42.965 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:42.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.952+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T09:56:42.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.952+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:42.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.952+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:56:42.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:42.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63e0018390 2021-03-09T09:56:43.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:43.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:43.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:43.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:44.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:44.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:45.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:45.133+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:45.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:45.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:45.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:45.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:46.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:46.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:46.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:46.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:47.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:47.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:47.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e007b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T09:56:47.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e007b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:47.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e007b9f0 secure :-1 s=READY pgs=11269 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:47.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e007b9f0 secure :-1 s=READY pgs=11269 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc003a50).stop 2021-03-09T09:56:47.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:47.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:47.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e007b9f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:47.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e007b9f0 unknown :-1 s=CLOSED pgs=11269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:47.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:56:47.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:47.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:47.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:47.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:47.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:47.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:47.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:47.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016db0 con 0x7f63d8003400 2021-03-09T09:56:47.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:47.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026010 con 0x7f63d8008840 2021-03-09T09:56:47.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:47.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c6f0 con 0x7f6400146050 2021-03-09T09:56:47.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:47.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:47.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.950+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:47.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.951+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:47.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.951+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:47.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:47.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:56:47.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:47.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:47.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:47.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:47.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:47.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:47.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:47.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:47.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:47.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:47.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:56:47.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:56:47.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:47.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:47.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:56:47.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:47.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:47.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00658a0 con 0x7f6400146050 2021-03-09T09:56:47.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:56:47.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.952+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:47.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.953+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021890 secure :-1 s=READY pgs=11278 cs=0 l=1 rev1=1 rx=0x7f63e0016aa0 tx=0x7f63fc052f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:47.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.953+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:56:47.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.953+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:47.967 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:47.954+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 30984 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:47.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06feb0 con 0x7f6400146050 2021-03-09T09:56:47.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.955+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:47.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.955+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:47.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.955+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:47.969 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:47.969 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:47.969 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:47.969 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:47.970 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:47.970 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:47.970 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:47.970 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:47.971 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:47.971 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:47.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.958+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06feb0 con 0x7f6400146050 2021-03-09T09:56:47.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.958+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:47.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.958+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:56:47.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:47.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04b7b0 con 0x7f6400146050 2021-03-09T09:56:48.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:48.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:48.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:48.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:48.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:48.612+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:49.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:49.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:49.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:49.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:50.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:50.134+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:50.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:50.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:50.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:50.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:51.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:51.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:51.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:51.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:52.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:52.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:52.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:52.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021890 secure :-1 s=READY pgs=11278 cs=0 l=1 rev1=1 rx=0x7f63e0016aa0 tx=0x7f63fc052f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:52.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021890 secure :-1 s=READY pgs=11278 cs=0 l=1 rev1=1 rx=0x7f63e0016aa0 tx=0x7f63fc052f50).stop 2021-03-09T09:56:52.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:52.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:52.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0021890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:52.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0021890 unknown :-1 s=CLOSED pgs=11278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:52.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:56:52.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:52.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:52.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:52.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:52.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:52.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:52.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:52.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e480 con 0x7f63d8003400 2021-03-09T09:56:52.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:52.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:56:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63e0018390 2021-03-09T09:56:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:52.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:52.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:52.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:52.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:52.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:56:52.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:52.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.955+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:52.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:52.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:52.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:52.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:52.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:52.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:52.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:52.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:52.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:56:52.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:56:52.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:52.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:52.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065710 con 0x7f63e0018390 2021-03-09T09:56:52.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:52.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:52.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T09:56:52.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:56:52.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.956+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:52.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.958+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:56:52.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.958+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:52.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.958+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11290 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4048580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:52.974 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:52.959+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31024 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:52.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.959+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:56:52.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.959+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:52.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.959+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:52.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.959+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:52.978 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:52.978 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:52.978 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:52.979 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:52.979 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:52.980 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:52.980 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:52.981 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:52.981 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:52.982 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:52.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:56:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.960+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:52.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.960+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:56:52.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:52.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40341b0 con 0x7f63e0018390 2021-03-09T09:56:53.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:53.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:53.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:53.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:54.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:54.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:54.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:54.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:55.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:55.135+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:56:55.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:55.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:55.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:55.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:56.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:56.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:56.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:56.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:57.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:57.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:57.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:56:57.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:56:57.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11290 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4048580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:56:57.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11290 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4048580).stop 2021-03-09T09:56:57.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:56:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:56:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:56:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:56:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:56:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:56:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:56:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:56:57.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:57.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3a0 con 0x7f63d8003400 2021-03-09T09:56:57.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:57.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016db0 con 0x7f63d8008840 2021-03-09T09:56:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:56:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f6400146050 2021-03-09T09:56:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:56:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:56:57.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:56:57.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:57.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:56:57.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:56:57.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:56:57.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:56:57.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.960+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:56:57.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:56:57.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:56:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:56:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:56:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:56:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:56:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:56:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:56:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:57.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:57.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce80 con 0x7f6400146050 2021-03-09T09:56:57.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:56:57.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:56:57.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065710 con 0x7f6400146050 2021-03-09T09:56:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.962+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:56:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.962+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.962+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000e7c0 secure :-1 s=READY pgs=11305 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:56:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.962+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:56:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.962+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:56:57.977 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:56:57.964+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31064 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:56:57.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f6400146050 2021-03-09T09:56:57.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.965+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:56:57.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.965+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:56:57.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.965+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:56:57.979 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:56:57.979 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:56:57.979 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:57.979 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:56:57.980 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:56:57.980 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:56:57.980 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:56:57.980 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:56:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:56:57.981 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:56:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f6400146050 2021-03-09T09:56:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.969+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:56:57.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.969+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:56:57.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:57.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f6400146050 2021-03-09T09:56:58.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:58.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:58.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:58.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:56:58.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:58.614+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:56:59.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:59.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:56:59.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:56:59.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:00.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:00.136+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:00.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:00.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:00.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:00.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:01.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:01.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:01.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:01.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:02.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:02.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:02.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:02.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:02.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000e7c0 secure :-1 s=READY pgs=11305 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003bb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:02.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000e7c0 secure :-1 s=READY pgs=11305 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003bb0).stop 2021-03-09T09:57:02.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:02.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:02.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e000e7c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:02.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e000e7c0 unknown :-1 s=CLOSED pgs=11305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:02.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:57:02.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:02.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:02.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:02.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:02.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:02.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:02.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:02.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028540 con 0x7f63d8003400 2021-03-09T09:57:02.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:02.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8008840 2021-03-09T09:57:02.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:02.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c6f0 con 0x7f63e0018390 2021-03-09T09:57:02.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.965+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.966+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.966+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:02.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.967+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:02.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.967+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:57:02.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.967+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:02.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.967+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:02.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.967+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:02.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.967+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:02.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:02.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:02.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:02.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:02.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:02.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:02.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:57:02.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:57:02.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002b8a0 con 0x7f63e0018390 2021-03-09T09:57:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017c90 con 0x7f63e0018390 2021-03-09T09:57:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:57:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.968+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.969+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:57:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.969+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11320 cs=0 l=1 rev1=1 rx=0x7f63f0074040 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.969+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:02.981 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:02.970+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31093 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:02.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00542c0 con 0x7f63e0018390 2021-03-09T09:57:02.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.971+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:02.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.971+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:02.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.971+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:02.982 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:02.983 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:02.983 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:02.983 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:02.983 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:02.984 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:02.984 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:02.984 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:02.984 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:02.984 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:02.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.974+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00157f0 con 0x7f63e0018390 2021-03-09T09:57:02.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.974+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:02.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.974+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:57:02.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:02.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005eb10 con 0x7f63e0018390 2021-03-09T09:57:03.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:03.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:03.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:03.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:04.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:04.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:04.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:04.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:05.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:05.137+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:05.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:05.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:05.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:05.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:06.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:06.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:06.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:06.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:07.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:07.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:07.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:07.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:07.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11320 cs=0 l=1 rev1=1 rx=0x7f63f0074040 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:07.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11320 cs=0 l=1 rev1=1 rx=0x7f63f0074040 tx=0x7f63f0000c30).stop 2021-03-09T09:57:07.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:07.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:07.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:07.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:07.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:57:07.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:07.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:07.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:07.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:07.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:07.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:07.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:07.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:57:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T09:57:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400146050 2021-03-09T09:57:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:07.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:07.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:07.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:07.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:57:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:07.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:07.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.971+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:07.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:57:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:57:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:57:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:57:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:57:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.972+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.973+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 secure :-1 s=READY pgs=11332 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.973+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:57:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.973+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:07.989 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:07.975+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31142 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:07.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc006550 con 0x7f6400146050 2021-03-09T09:57:07.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.976+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.976+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.976+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:07.992 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:07.992 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:07.992 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:07.992 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:07.993 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:07.993 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:07.993 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:07.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc006550 con 0x7f6400146050 2021-03-09T09:57:07.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.976+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:07.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.976+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:57:07.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:07.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T09:57:08.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:08.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:08.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:08.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:08.614+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:09.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:09.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:09.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:09.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:10.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:10.138+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:10.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:10.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:10.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:10.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:11.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:11.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:11.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:11.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:12.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:12.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:12.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:12.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:12.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 secure :-1 s=READY pgs=11332 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:12.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 secure :-1 s=READY pgs=11332 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04a6d0).stop 2021-03-09T09:57:12.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:12.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:12.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:12.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0023390 unknown :-1 s=CLOSED pgs=11332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:12.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:57:12.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:12.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:12.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:12.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:12.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:12.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:12.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:12.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e480 con 0x7f63d8003400 2021-03-09T09:57:12.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:12.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:57:12.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:12.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e0018390 2021-03-09T09:57:12.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.975+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:12.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.976+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:12.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.976+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:12.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:12.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.976+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:12.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:12.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:57:12.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:12.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:12.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:12.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:12.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:12.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:12.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:12.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:12.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:12.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:12.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:57:12.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:57:12.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:12.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:12.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:57:12.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:12.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.977+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:12.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.978+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce80 con 0x7f63e0018390 2021-03-09T09:57:12.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.978+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:57:12.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.978+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:12.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.978+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11346 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4053af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:12.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.978+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:57:12.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.978+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:12.992 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:12.980+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31182 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:12.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4046e40 con 0x7f63e0018390 2021-03-09T09:57:12.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.981+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:12.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.981+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:12.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.981+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:12.994 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:12.994 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:12.995 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:12.995 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:12.995 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:12.995 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:12.995 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:12.996 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:12.996 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:12.996 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:12.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.985+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046e40 con 0x7f63e0018390 2021-03-09T09:57:12.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.985+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:12.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.985+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:57:12.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:12.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404cbc0 con 0x7f63e0018390 2021-03-09T09:57:13.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:13.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:13.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:13.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:14.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:14.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:14.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:14.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:15.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:15.139+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:15.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:15.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:15.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:15.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:16.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:16.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:16.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:16.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:17.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:17.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:17.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:17.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:17.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11346 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4053af0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:17.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11346 cs=0 l=1 rev1=1 rx=0x7f63f40052c0 tx=0x7f63f4053af0).stop 2021-03-09T09:57:17.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:17.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:17.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:17.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:17.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:57:17.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:17.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:17.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:17.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:17.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:17.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:17.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:17.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T09:57:17.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:17.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T09:57:17.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:17.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013f40 con 0x7f6400146050 2021-03-09T09:57:17.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:17.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:17.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:17.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:17.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.981+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:17.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.982+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:17.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.982+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:57:17.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.982+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:17.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.982+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:17.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.982+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:17.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.982+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:17.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.983+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:17.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:17.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:17.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:17.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:17.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:17.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:57:17.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:57:17.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:17.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:17.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00682d0 con 0x7f6400146050 2021-03-09T09:57:17.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:17.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:17.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:57:17.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:57:17.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.984+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:17.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.985+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:57:17.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.985+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11365 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc012ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:17.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.985+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:17.997 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:17.985+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31220 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:17.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012ca0 con 0x7f6400146050 2021-03-09T09:57:17.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.986+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:17.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.986+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.986+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:18.000 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:18.000 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:18.000 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:18.001 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:18.001 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:18.001 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:18.001 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:18.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc012a70 con 0x7f6400146050 2021-03-09T09:57:18.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.987+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:18.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.987+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:57:18.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:17.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T09:57:18.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:18.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:18.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:18.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:18.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:18.616+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:19.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:19.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:19.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:19.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:20.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:20.140+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:20.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:20.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:20.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:20.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:21.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:21.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:21.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:21.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:22.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:22.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:22.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11365 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc012ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11365 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc012ed0).stop 2021-03-09T09:57:22.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:22.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:22.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:22.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:22.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:57:22.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:22.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:22.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:22.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:22.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:22.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:22.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:22.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:57:22.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:22.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:57:22.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:22.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016980 con 0x7f63e0018390 2021-03-09T09:57:22.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:22.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:22.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.986+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:22.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.987+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:22.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.987+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:22.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:22.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.987+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:57:22.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.987+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:22.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.987+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:22.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.988+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:22.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.988+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:22.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.988+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:22.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:22.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:22.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:22.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:22.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:22.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:57:22.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:57:23.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:23.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:23.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f63e0018390 2021-03-09T09:57:23.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:23.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:23.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002b8a0 con 0x7f63e0018390 2021-03-09T09:57:23.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:57:23.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:23.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:57:23.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:23.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.989+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11378 cs=0 l=1 rev1=1 rx=0x7f63e0021360 tx=0x7f63f0054260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:23.003 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:22.990+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31250 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:23.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005edf0 con 0x7f63e0018390 2021-03-09T09:57:23.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.991+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:23.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.991+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:23.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.991+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:23.004 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:23.004 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:23.005 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:23.005 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:23.005 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:23.005 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:23.006 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:23.006 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:23.006 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:23.006 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:23.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005edf0 con 0x7f63e0018390 2021-03-09T09:57:23.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.991+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:23.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.991+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:57:23.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:22.994+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001cc80 con 0x7f63e0018390 2021-03-09T09:57:23.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:23.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:23.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:23.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:24.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:24.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:24.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:24.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:25.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:25.141+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:25.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:25.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:25.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:25.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:26.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:26.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:26.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:26.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:27.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:27.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:27.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:27.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:27.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11378 cs=0 l=1 rev1=1 rx=0x7f63e0021360 tx=0x7f63f0054260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:27.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11378 cs=0 l=1 rev1=1 rx=0x7f63e0021360 tx=0x7f63f0054260).stop 2021-03-09T09:57:27.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:27.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:27.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:27.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:57:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:27.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:28.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:28.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:28.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:28.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068190 con 0x7f63d8003400 2021-03-09T09:57:28.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:28.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d650 con 0x7f63d8008840 2021-03-09T09:57:28.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:28.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400146050 2021-03-09T09:57:28.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:28.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:28.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:28.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.991+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:28.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.992+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:57:28.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.992+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:28.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.992+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:28.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.992+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:28.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.992+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:28.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.992+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:28.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:28.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:28.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:28.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:28.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:28.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:57:28.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:57:28.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:28.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:28.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T09:57:28.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:28.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:28.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f6400146050 2021-03-09T09:57:28.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.994+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:57:28.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.994+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:28.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.994+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11389 cs=0 l=1 rev1=1 rx=0x7f63f0054010 tx=0x7f63f007ff30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:28.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.994+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:57:28.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.994+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:28.019 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:27.995+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31298 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:28.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0088f00 con 0x7f6400146050 2021-03-09T09:57:28.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.996+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:28.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.996+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:28.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.996+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:28.022 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:28.022 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:28.023 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:28.023 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:28.023 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:28.024 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:28.024 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:28.025 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:28.025 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:28.026 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:28.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f6400146050 2021-03-09T09:57:28.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.996+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:28.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.997+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:57:28.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:27.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027460 con 0x7f6400146050 2021-03-09T09:57:28.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:28.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:28.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:28.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:28.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:28.618+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:29.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:29.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:29.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:29.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:30.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:30.142+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:30.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:30.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:30.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:30.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:31.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:31.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:31.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:31.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:32.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:32.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:32.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:32.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:32.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11389 cs=0 l=1 rev1=1 rx=0x7f63f0054010 tx=0x7f63f007ff30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:33.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11389 cs=0 l=1 rev1=1 rx=0x7f63f0054010 tx=0x7f63f007ff30).stop 2021-03-09T09:57:33.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:33.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:33.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:33.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:33.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:57:33.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:33.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:33.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:33.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:33.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:33.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:33.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:33.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026010 con 0x7f63d8003400 2021-03-09T09:57:33.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:33.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017480 con 0x7f63d8008840 2021-03-09T09:57:33.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:33.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e480 con 0x7f63e0018390 2021-03-09T09:57:33.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:33.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:33.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:33.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:33.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:33.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.998+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:33.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.998+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:57:33.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.998+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:33.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.998+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:33.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.998+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:33.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.998+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:33.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.998+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:33.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:33.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:33.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:33.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:33.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:33.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:57:33.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:57:33.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:33.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:33.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002b8a0 con 0x7f63e0018390 2021-03-09T09:57:33.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:33.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:33.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00682d0 con 0x7f63e0018390 2021-03-09T09:57:33.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:57:33.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:32.999+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:33.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.001+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11404 cs=0 l=1 rev1=1 rx=0x7f63e00217c0 tx=0x7f63fc000f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:33.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.001+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:57:33.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.001+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:33.012 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:33.001+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31337 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:33.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03dd90 con 0x7f63e0018390 2021-03-09T09:57:33.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.002+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:33.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.002+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:33.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.002+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:33.013 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:33.014 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:33.014 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:33.014 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:33.014 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:33.015 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:33.015 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:33.015 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:33.015 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:33.015 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:33.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03dd90 con 0x7f63e0018390 2021-03-09T09:57:33.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.006+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:33.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.006+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:57:33.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63e0018390 2021-03-09T09:57:33.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:33.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:33.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:34.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:34.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:34.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:34.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:35.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:35.143+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:35.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:35.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:35.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:35.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:36.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:36.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:36.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:36.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:37.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:37.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:37.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:37.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:38.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:38.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:38.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11404 cs=0 l=1 rev1=1 rx=0x7f63e00217c0 tx=0x7f63fc000f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:38.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11404 cs=0 l=1 rev1=1 rx=0x7f63e00217c0 tx=0x7f63fc000f40).stop 2021-03-09T09:57:38.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:38.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:38.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:57:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:38.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:38.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:57:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013880 con 0x7f63d8008840 2021-03-09T09:57:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:57:38.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:38.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:38.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.002+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:38.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:38.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.003+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:38.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.003+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:38.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.003+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:57:38.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.003+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:38.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.003+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:38.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.004+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:38.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.004+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:38.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.004+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:38.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:57:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:57:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:38.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce80 con 0x7f6400146050 2021-03-09T09:57:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002b8a0 con 0x7f6400146050 2021-03-09T09:57:38.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:57:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.005+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.006+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11423 cs=0 l=1 rev1=1 rx=0x7f63fc071360 tx=0x7f63fc06de70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.006+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:57:38.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.006+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:38.016 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:38.006+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31376 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:38.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012a70 con 0x7f6400146050 2021-03-09T09:57:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.007+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.007+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.007+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:38.017 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:38.018 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:38.018 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:38.018 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:38.018 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:38.019 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:38.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc012a70 con 0x7f6400146050 2021-03-09T09:57:38.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.012+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:38.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.012+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:57:38.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T09:57:38.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:38.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:38.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:38.619+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:39.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:39.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:39.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:39.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:40.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:40.144+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:40.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:40.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:40.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:40.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:41.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:41.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:41.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:41.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:42.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:42.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:42.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:42.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:43.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.007+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:43.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.007+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:43.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.007+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11423 cs=0 l=1 rev1=1 rx=0x7f63fc071360 tx=0x7f63fc06de70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:43.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.007+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11423 cs=0 l=1 rev1=1 rx=0x7f63fc071360 tx=0x7f63fc06de70).stop 2021-03-09T09:57:43.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.007+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:43.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.007+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:43.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:43.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.007+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:43.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:57:43.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:43.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:43.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:43.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:43.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:43.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:43.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:43.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067910 con 0x7f63d8003400 2021-03-09T09:57:43.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:43.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f63d8008840 2021-03-09T09:57:43.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:43.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f040 con 0x7f63e0018390 2021-03-09T09:57:43.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:43.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:43.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:43.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:43.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:43.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:43.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:57:43.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:43.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.008+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:43.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:43.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:43.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:43.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:43.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:43.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:43.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:43.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:43.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:57:43.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:57:43.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:43.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:43.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005de90 con 0x7f63e0018390 2021-03-09T09:57:43.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:43.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:43.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T09:57:43.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:57:43.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.009+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:43.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.010+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:57:43.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.010+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:43.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.010+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11430 cs=0 l=1 rev1=1 rx=0x7f63f0027c40 tx=0x7f63f0030320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:43.030 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:43.012+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31414 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:43.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63e0018390 2021-03-09T09:57:43.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.013+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:43.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.013+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:43.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.013+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:43.031 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:43.032 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:43.032 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:43.032 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:43.032 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:43.032 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:43.033 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:43.033 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:43.033 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:43.033 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:43.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027d60 con 0x7f63e0018390 2021-03-09T09:57:43.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.016+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:43.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.016+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:57:43.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e290 con 0x7f63e0018390 2021-03-09T09:57:43.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:43.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:43.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:44.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:44.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:44.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:44.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:45.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:45.145+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:45.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:45.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:45.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:45.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:46.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:46.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:46.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:46.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:47.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:47.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:48.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:57:48.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:48.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11430 cs=0 l=1 rev1=1 rx=0x7f63f0027c40 tx=0x7f63f0030320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:48.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11430 cs=0 l=1 rev1=1 rx=0x7f63f0027c40 tx=0x7f63f0030320).stop 2021-03-09T09:57:48.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:48.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:48.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:48.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:48.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:57:48.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:48.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:48.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:48.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:48.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:48.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:48.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:48.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e480 con 0x7f63d8003400 2021-03-09T09:57:48.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:48.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8008840 2021-03-09T09:57:48.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:48.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:57:48.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:48.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:48.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:48.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:48.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:57:48.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:48.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:48.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:48.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:48.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:48.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:48.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:48.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:48.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:48.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:48.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:48.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:48.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:57:48.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:57:48.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:48.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:48.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000f6c0 con 0x7f6400146050 2021-03-09T09:57:48.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:48.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:48.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:57:48.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:57:48.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.014+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:48.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.015+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11440 cs=0 l=1 rev1=1 rx=0x7f63f402dfd0 tx=0x7f63f4046f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:48.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.015+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:57:48.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.015+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:48.027 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:48.017+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31454 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:48.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f6400146050 2021-03-09T09:57:48.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.018+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:48.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.018+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:48.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.018+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:48.029 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:48.029 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:48.030 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:48.030 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:48.030 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:48.030 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:48.031 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:48.031 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:48.031 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:48.031 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:48.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f6400146050 2021-03-09T09:57:48.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.019+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:48.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.019+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:57:48.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f6400146050 2021-03-09T09:57:48.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:48.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:48.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:48.621+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:49.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:49.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:49.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:49.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:50.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:50.146+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:50.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:50.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:50.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:50.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:51.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:51.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:51.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:51.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:52.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:52.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:52.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:52.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:53.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:53.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:53.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11440 cs=0 l=1 rev1=1 rx=0x7f63f402dfd0 tx=0x7f63f4046f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:53.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11440 cs=0 l=1 rev1=1 rx=0x7f63f402dfd0 tx=0x7f63f4046f20).stop 2021-03-09T09:57:53.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:53.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:53.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:53.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:53.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:57:53.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:53.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:53.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:53.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:53.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:53.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:53.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:53.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026750 con 0x7f63d8003400 2021-03-09T09:57:53.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:53.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T09:57:53.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:53.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f040 con 0x7f63e0018390 2021-03-09T09:57:53.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:53.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:53.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:53.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:53.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.018+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:53.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.019+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:53.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.019+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:57:53.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.019+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:53.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.019+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:53.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.019+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:53.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.019+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:53.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.019+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:53.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:53.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:53.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:53.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:53.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:53.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:57:53.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:57:53.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:53.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:53.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:57:53.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:53.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:53.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000f6c0 con 0x7f63e0018390 2021-03-09T09:57:53.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:57:53.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.020+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:53.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.021+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11458 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f400ee80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:53.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.021+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:57:53.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.021+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:53.033 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:53.023+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31492 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:53.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022a80 con 0x7f63e0018390 2021-03-09T09:57:53.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.024+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:53.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.024+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:53.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.024+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:53.035 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:53.035 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:53.035 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:53.035 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:53.035 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:53.036 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:53.036 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:53.036 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:53.036 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:53.036 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:53.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022be0 con 0x7f63e0018390 2021-03-09T09:57:53.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.024+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:53.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.024+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:57:53.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f401a270 con 0x7f63e0018390 2021-03-09T09:57:53.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:53.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:53.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:54.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:54.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:54.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:54.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:55.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:55.146+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:57:55.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:55.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:55.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:55.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:56.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:56.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:56.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:56.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:57.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:57.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:57.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:57.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:58.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:57:58.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:57:58.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11458 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f400ee80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:57:58.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11458 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f400ee80).stop 2021-03-09T09:57:58.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:58.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:57:58.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:57:58.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:57:58.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:57:58.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:58.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:57:58.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:58.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:57:58.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:57:58.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:57:58.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:58.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:57:58.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:58.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022f80 con 0x7f63d8008840 2021-03-09T09:57:58.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:57:58.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064d10 con 0x7f6400146050 2021-03-09T09:57:58.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:57:58.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:57:58.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:58.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:57:58.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.023+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:57:58.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:57:58.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:57:58.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:57:58.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:57:58.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:57:58.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:57:58.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:57:58.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:57:58.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:58.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:58.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:57:58.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:57:58.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:57:58.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:57:58.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:58.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:58.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016190 con 0x7f6400146050 2021-03-09T09:57:58.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:57:58.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:57:58.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce80 con 0x7f6400146050 2021-03-09T09:57:58.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:57:58.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.024+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:58.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11467 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:57:58.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.025+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:57:58.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.025+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:57:58.038 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:57:58.027+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31532 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:57:58.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013c60 con 0x7f6400146050 2021-03-09T09:57:58.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.028+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:57:58.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.028+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:57:58.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.028+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:57:58.040 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:57:58.041 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:57:58.041 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:58.041 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:57:58.041 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:57:58.041 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:57:58.042 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:57:58.042 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:57:58.042 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:57:58.042 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:57:58.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013c60 con 0x7f6400146050 2021-03-09T09:57:58.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.033+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:57:58.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.033+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:57:58.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.034+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T09:57:58.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:58.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:57:58.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:58.621+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:57:59.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:59.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:57:59.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:57:59.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:00.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:00.147+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:00.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:00.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:00.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:00.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:01.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:01.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:01.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:01.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:02.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:02.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:02.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:02.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:03.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.027+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:03.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:03.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11467 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:03.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11467 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc009a70).stop 2021-03-09T09:58:03.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:03.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:03.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:03.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:03.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:58:03.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:03.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:03.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:03.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:03.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:03.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:03.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:03.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f040 con 0x7f63d8003400 2021-03-09T09:58:03.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:03.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:58:03.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:03.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e480 con 0x7f63e0018390 2021-03-09T09:58:03.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:03.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:03.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:03.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:03.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:03.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:03.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:58:03.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:03.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.028+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:03.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.029+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:03.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.029+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:03.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.029+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:03.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:03.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:03.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:03.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:03.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:03.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:58:03.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:58:03.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:03.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:03.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f63e0018390 2021-03-09T09:58:03.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:03.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:03.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016190 con 0x7f63e0018390 2021-03-09T09:58:03.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:58:03.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.030+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:03.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.031+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:58:03.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.031+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11483 cs=0 l=1 rev1=1 rx=0x7f63fc047e30 tx=0x7f63fc058610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:03.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.031+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:03.046 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:03.032+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31570 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:03.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058a70 con 0x7f63e0018390 2021-03-09T09:58:03.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.033+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.033+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.033+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:03.048 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:03.049 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:03.049 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:03.049 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:03.050 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:03.051 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.034+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058a70 con 0x7f63e0018390 2021-03-09T09:58:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.034+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:03.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.034+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:58:03.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.034+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04b7b0 con 0x7f63e0018390 2021-03-09T09:58:03.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:03.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:03.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:04.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:04.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:05.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:05.148+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:05.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:05.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:05.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:05.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:06.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:06.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:06.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:06.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:07.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:07.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:07.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:07.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:08.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.032+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:08.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.032+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:08.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.032+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11483 cs=0 l=1 rev1=1 rx=0x7f63fc047e30 tx=0x7f63fc058610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:08.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11483 cs=0 l=1 rev1=1 rx=0x7f63fc047e30 tx=0x7f63fc058610).stop 2021-03-09T09:58:08.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:08.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:08.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:08.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:08.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:58:08.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:08.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:08.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:08.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:08.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:08.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:08.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:08.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T09:58:08.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:08.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8008840 2021-03-09T09:58:08.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:08.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f6400146050 2021-03-09T09:58:08.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:08.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:08.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:08.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:08.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:08.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.033+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:08.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.034+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:58:08.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.034+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:08.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.034+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:08.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.034+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:08.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.034+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:08.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.034+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:08.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:08.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:08.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:08.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:08.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:08.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:58:08.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:58:08.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:08.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:08.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ccf0 con 0x7f6400146050 2021-03-09T09:58:08.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:08.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:08.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f6400146050 2021-03-09T09:58:08.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:58:08.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.035+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:08.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.036+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:58:08.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.036+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:08.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.036+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11499 cs=0 l=1 rev1=1 rx=0x7f63e0024b70 tx=0x7f63fc078e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:08.048 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:08.038+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31610 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:08.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f6400146050 2021-03-09T09:58:08.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.039+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:08.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.039+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:08.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.039+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:08.050 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:08.050 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:08.050 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:08.051 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:08.051 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:08.051 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:08.051 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:08.051 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:08.052 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:08.052 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:08.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058a70 con 0x7f6400146050 2021-03-09T09:58:08.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.044+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:08.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04b7b0 con 0x7f6400146050 2021-03-09T09:58:08.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.044+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:58:08.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:08.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:08.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:08.622+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:09.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:09.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:09.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:09.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:10.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:10.149+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:10.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:10.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:10.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:10.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:11.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:11.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:11.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:11.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:12.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:12.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:12.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:13.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:13.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:13.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11499 cs=0 l=1 rev1=1 rx=0x7f63e0024b70 tx=0x7f63fc078e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:13.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11499 cs=0 l=1 rev1=1 rx=0x7f63e0024b70 tx=0x7f63fc078e30).stop 2021-03-09T09:58:13.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:13.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:13.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:13.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:13.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:58:13.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:13.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:13.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:13.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:13.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:13.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:13.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:13.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024260 con 0x7f63d8003400 2021-03-09T09:58:13.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:13.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:58:13.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:13.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f910 con 0x7f63e0018390 2021-03-09T09:58:13.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:13.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:13.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.038+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:13.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.039+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:13.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.039+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:13.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.039+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:13.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.039+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:58:13.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.039+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:13.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.039+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:13.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:13.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:13.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:13.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:13.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:13.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:13.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:13.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:13.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:58:13.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:58:13.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:13.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:13.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0018390 2021-03-09T09:58:13.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:13.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:13.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.040+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:58:13.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.041+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:58:13.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.041+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:13.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.041+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11511 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4005ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:13.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.041+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:58:13.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.041+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:13.054 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:13.043+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31648 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:13.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:58:13.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.044+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:13.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.044+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:13.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.044+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:13.056 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:13.056 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:13.056 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:13.057 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:13.057 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:13.057 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:13.057 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:13.057 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:13.058 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:13.058 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:13.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:58:13.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.051+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:13.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.051+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:58:13.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404cbc0 con 0x7f63e0018390 2021-03-09T09:58:13.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:13.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:13.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:14.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:14.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:14.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:14.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:15.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:15.150+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:15.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:15.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:15.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:15.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:16.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:16.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:16.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:16.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:17.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:17.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:17.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:17.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:18.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:18.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:18.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11511 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4005ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:18.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11511 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4005ea0).stop 2021-03-09T09:58:18.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:18.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:18.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:18.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:18.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:58:18.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:18.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:18.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:18.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:18.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:18.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:18.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:18.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00155c0 con 0x7f63d8003400 2021-03-09T09:58:18.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:18.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001420 con 0x7f63d8008840 2021-03-09T09:58:18.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:18.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f6400146050 2021-03-09T09:58:18.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:18.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:18.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:18.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:18.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.044+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:18.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.045+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:18.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.045+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:58:18.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.045+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:18.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.045+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.045+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.045+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:18.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.045+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:18.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:18.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:18.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:18.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:18.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:18.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:58:18.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:58:18.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:18.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:18.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce80 con 0x7f6400146050 2021-03-09T09:58:18.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:18.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:18.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:58:18.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:58:18.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:18.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11527 cs=0 l=1 rev1=1 rx=0x7f63f4072e20 tx=0x7f63f404cfa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:18.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:58:18.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.047+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:18.062 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:18.048+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31688 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:18.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072e90 con 0x7f6400146050 2021-03-09T09:58:18.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.049+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:18.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.049+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:18.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.049+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:18.065 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:18.066 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:18.066 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:18.067 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:18.067 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:18.068 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:18.068 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:18.068 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:18.069 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:18.069 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:18.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072c60 con 0x7f6400146050 2021-03-09T09:58:18.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.049+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:18.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.049+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:58:18.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022a80 con 0x7f6400146050 2021-03-09T09:58:18.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:18.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:18.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:18.624+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:19.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:19.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:19.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:19.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:20.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:20.151+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:20.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:20.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:20.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:20.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:21.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:21.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:21.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:21.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:22.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:22.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:22.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:22.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:23.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:23.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:23.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11527 cs=0 l=1 rev1=1 rx=0x7f63f4072e20 tx=0x7f63f404cfa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:23.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11527 cs=0 l=1 rev1=1 rx=0x7f63f4072e20 tx=0x7f63f404cfa0).stop 2021-03-09T09:58:23.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:23.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:23.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:23.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:23.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:58:23.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:23.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:23.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00254f0 con 0x7f63d8003400 2021-03-09T09:58:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:58:23.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027c60 con 0x7f63e0018390 2021-03-09T09:58:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:23.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:23.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.049+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:23.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:23.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.050+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:58:23.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.050+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:23.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.050+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:23.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.050+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:23.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.050+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.050+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:23.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:58:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:58:23.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:23.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:23.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T09:58:23.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:23.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:23.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005de90 con 0x7f63e0018390 2021-03-09T09:58:23.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:58:23.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.051+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:23.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.052+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:58:23.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.053+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:23.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11542 cs=0 l=1 rev1=1 rx=0x7f63e0048e80 tx=0x7f63f0015860).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:23.065 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:23.054+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31726 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00542c0 con 0x7f63e0018390 2021-03-09T09:58:23.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.055+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:23.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.055+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:23.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.055+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:23.067 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:23.067 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:23.068 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:23.068 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:23.068 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:23.068 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:23.068 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:23.069 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:23.069 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:23.069 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:23.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0088ec0 con 0x7f63e0018390 2021-03-09T09:58:23.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.059+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:23.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.059+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:58:23.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005eb10 con 0x7f63e0018390 2021-03-09T09:58:23.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:23.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:23.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:24.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:24.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:24.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:24.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:25.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:25.152+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:25.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:25.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:25.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:25.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:26.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:26.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:26.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:26.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:27.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:27.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:27.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:27.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:28.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:28.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:28.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11542 cs=0 l=1 rev1=1 rx=0x7f63e0048e80 tx=0x7f63f0015860).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:28.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11542 cs=0 l=1 rev1=1 rx=0x7f63e0048e80 tx=0x7f63f0015860).stop 2021-03-09T09:58:28.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:28.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:28.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:28.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:28.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T09:58:28.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:28.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:28.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:28.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:28.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:28.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:28.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:28.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024260 con 0x7f63d8003400 2021-03-09T09:58:28.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:28.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cd10 con 0x7f63d8008840 2021-03-09T09:58:28.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:28.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:58:28.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:28.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:28.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.054+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:28.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.055+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:28.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.055+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:28.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:28.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:58:28.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:28.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:28.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:28.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:28.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:28.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:28.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:28.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:28.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:28.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:28.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:58:28.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:58:28.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:28.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:28.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:58:28.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:28.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:28.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da60 con 0x7f6400146050 2021-03-09T09:58:28.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:58:28.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.056+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:28.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.057+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:58:28.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.057+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:28.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.057+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11554 cs=0 l=1 rev1=1 rx=0x7f63f0032010 tx=0x7f63f00282a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:28.070 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:28.059+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31766 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:28.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.061+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028830 con 0x7f6400146050 2021-03-09T09:58:28.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.061+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:28.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.061+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:28.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.061+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:28.072 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:28.072 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:28.072 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:28.072 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:28.072 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:28.073 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:28.073 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:28.073 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:28.073 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:28.074 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:28.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028830 con 0x7f6400146050 2021-03-09T09:58:28.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.063+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:28.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.063+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:58:28.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00036b0 con 0x7f6400146050 2021-03-09T09:58:28.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:28.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:28.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:28.625+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:29.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:29.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:29.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:29.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:30.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:30.153+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:30.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:30.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:30.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:30.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:31.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:31.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:31.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:31.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:32.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:32.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:32.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:32.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:33.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:33.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:33.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11554 cs=0 l=1 rev1=1 rx=0x7f63f0032010 tx=0x7f63f00282a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:33.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11554 cs=0 l=1 rev1=1 rx=0x7f63f0032010 tx=0x7f63f00282a0).stop 2021-03-09T09:58:33.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:33.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:33.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:33.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:33.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:58:33.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:33.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:33.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:33.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:33.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:33.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:33.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:33.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019480 con 0x7f63d8003400 2021-03-09T09:58:33.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:33.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3a0 con 0x7f63d8008840 2021-03-09T09:58:33.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:33.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e150 con 0x7f63e0018390 2021-03-09T09:58:33.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:33.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:33.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:33.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.060+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:33.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.060+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:33.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.060+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:33.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.060+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:58:33.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.060+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:33.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.060+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:33.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.060+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:33.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.060+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:33.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.061+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:33.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:33.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:58:33.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:58:33.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:33.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:33.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:58:33.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:33.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:33.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce80 con 0x7f63e0018390 2021-03-09T09:58:33.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:58:33.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:33.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11569 cs=0 l=1 rev1=1 rx=0x7f63f406e040 tx=0x7f63f405bfa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:33.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:58:33.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.062+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:33.076 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:33.063+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31800 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:33.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:58:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.064+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.064+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.064+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:33.077 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:33.078 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:33.079 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:33.080 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:33.080 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:33.080 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:33.081 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:33.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T09:58:33.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.068+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:33.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.068+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:58:33.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404b270 con 0x7f63e0018390 2021-03-09T09:58:33.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:33.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:33.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:34.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:34.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:34.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:34.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:35.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:35.154+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:35.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:35.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:35.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:35.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:36.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:36.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:37.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:37.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:37.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:37.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:38.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:58:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11569 cs=0 l=1 rev1=1 rx=0x7f63f406e040 tx=0x7f63f405bfa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:38.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11569 cs=0 l=1 rev1=1 rx=0x7f63f406e040 tx=0x7f63f405bfa0).stop 2021-03-09T09:58:38.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:38.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:38.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:38.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:38.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:58:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:38.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:38.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:38.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:38.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T09:58:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T09:58:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:38.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021450 con 0x7f6400146050 2021-03-09T09:58:38.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:38.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:38.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.064+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:38.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.065+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:38.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.065+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:38.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.065+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:38.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.065+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:58:38.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.065+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.065+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.065+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.065+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:38.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:38.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:38.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:38.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:38.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:38.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:58:38.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:58:38.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:38.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:38.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T09:58:38.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:38.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:38.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:58:38.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:58:38.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.066+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:38.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.068+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11587 cs=0 l=1 rev1=1 rx=0x7f63f404a040 tx=0x7f63f406a710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:38.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.068+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:58:38.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.068+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:38.081 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:38.069+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31844 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:38.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404de30 con 0x7f6400146050 2021-03-09T09:58:38.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.070+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:38.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.070+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:38.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.070+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:38.083 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:38.083 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:38.083 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:38.084 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:38.085 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404de30 con 0x7f6400146050 2021-03-09T09:58:38.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.075+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:38.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f6400146050 2021-03-09T09:58:38.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.075+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:58:38.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:38.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:38.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:38.626+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:39.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:39.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:39.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:39.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:40.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:40.155+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:40.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:40.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:40.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:40.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:41.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:41.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:41.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:41.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:42.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:42.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:42.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:42.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:43.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.069+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:43.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.069+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:43.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.069+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11587 cs=0 l=1 rev1=1 rx=0x7f63f404a040 tx=0x7f63f406a710).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:43.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.069+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11587 cs=0 l=1 rev1=1 rx=0x7f63f404a040 tx=0x7f63f406a710).stop 2021-03-09T09:58:43.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.069+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:43.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.069+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:43.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:43.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.069+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:43.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:58:43.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:43.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:43.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:43.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:43.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:43.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:43.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:43.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:58:43.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:43.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bf10 con 0x7f63d8008840 2021-03-09T09:58:43.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:43.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001420 con 0x7f63e0018390 2021-03-09T09:58:43.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:43.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:43.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:43.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:43.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.070+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:43.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.071+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:43.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.071+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:58:43.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.071+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:43.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.071+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:43.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.072+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:43.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.072+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:43.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.072+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:43.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:43.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:43.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:43.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:43.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:43.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:58:43.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:58:43.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:43.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:43.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016190 con 0x7f63e0018390 2021-03-09T09:58:43.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:43.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:43.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:58:43.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:58:43.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:43.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:58:43.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11598 cs=0 l=1 rev1=1 rx=0x7f63f0015930 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:43.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.073+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:43.099 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:43.074+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31882 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:43.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T09:58:43.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.075+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:43.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.075+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:43.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.075+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:43.103 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:43.103 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:43.103 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:43.104 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:43.104 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:43.105 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:43.105 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:43.106 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:43.106 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:43.107 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:43.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T09:58:43.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.079+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:43.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.080+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:58:43.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038770 con 0x7f63e0018390 2021-03-09T09:58:43.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:43.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:43.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:44.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:44.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:44.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:44.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:45.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:45.156+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:45.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:45.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:45.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:46.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:46.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:46.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:46.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:47.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:47.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:47.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:47.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:48.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:48.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:48.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11598 cs=0 l=1 rev1=1 rx=0x7f63f0015930 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:48.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11598 cs=0 l=1 rev1=1 rx=0x7f63f0015930 tx=0x7f63f004d180).stop 2021-03-09T09:58:48.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:48.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:48.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:48.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:48.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:58:48.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:48.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:48.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:48.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:48.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3a0 con 0x7f63d8003400 2021-03-09T09:58:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:48.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068390 con 0x7f63d8008840 2021-03-09T09:58:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e7f0 con 0x7f6400146050 2021-03-09T09:58:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:48.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:48.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:48.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.075+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:48.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.076+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:48.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.076+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:58:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.076+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.076+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.076+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.076+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:48.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.076+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:48.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:58:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:58:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:48.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:58:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T09:58:48.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:58:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:58:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:48.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.078+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11612 cs=0 l=1 rev1=1 rx=0x7f63f404bc90 tx=0x7f63f40099f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:48.090 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:48.079+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31922 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:48.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404de30 con 0x7f6400146050 2021-03-09T09:58:48.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.079+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:48.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.079+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:48.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.079+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:48.091 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:48.091 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:48.092 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:48.092 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:48.092 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:48.092 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:48.092 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:48.093 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.084+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f404de30 con 0x7f6400146050 2021-03-09T09:58:48.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.084+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:48.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.084+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f406a2d0 con 0x7f6400146050 2021-03-09T09:58:48.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.084+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:58:48.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:48.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:48.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:48.627+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:49.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:49.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:49.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:49.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:50.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:50.157+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:50.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:50.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:50.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:50.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:51.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:51.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:51.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:51.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:52.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:52.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:52.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:52.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:53.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:53.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:53.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11612 cs=0 l=1 rev1=1 rx=0x7f63f404bc90 tx=0x7f63f40099f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:53.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11612 cs=0 l=1 rev1=1 rx=0x7f63f404bc90 tx=0x7f63f40099f0).stop 2021-03-09T09:58:53.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:53.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:53.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:53.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:53.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:58:53.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:53.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:53.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:53.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:53.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:53.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:53.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:53.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ecb0 con 0x7f63d8003400 2021-03-09T09:58:53.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:53.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cf70 con 0x7f63d8008840 2021-03-09T09:58:53.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:53.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:58:53.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:53.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:53.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:53.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:53.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.080+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:53.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.081+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:53.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.081+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:58:53.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.081+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:53.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.081+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:53.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.082+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:53.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.082+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:53.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.082+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:53.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:53.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:53.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:53.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:53.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:53.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:58:53.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:58:53.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:53.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:53.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:58:53.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:53.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:53.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:58:53.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:58:53.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:53.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:58:53.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.084+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:53.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.083+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11630 cs=0 l=1 rev1=1 rx=0x7f63f405c070 tx=0x7f63f400e670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:53.101 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:53.084+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 31960 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:53.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.085+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404bd00 con 0x7f63e0018390 2021-03-09T09:58:53.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.085+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:53.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.085+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:53.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.085+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:53.103 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:53.103 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:53.104 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:53.104 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:53.104 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:53.104 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:53.104 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:53.105 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:53.105 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:53.105 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:53.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400e260 con 0x7f63e0018390 2021-03-09T09:58:53.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.089+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:53.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.089+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:58:53.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.090+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T09:58:53.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:53.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:53.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:54.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:54.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:54.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:54.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:55.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:55.158+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:58:55.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:55.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:55.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:56.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:56.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:56.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:56.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:57.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:57.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:57.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:57.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:58.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.084+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:58:58.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.084+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:58:58.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.084+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11630 cs=0 l=1 rev1=1 rx=0x7f63f405c070 tx=0x7f63f400e670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:58:58.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11630 cs=0 l=1 rev1=1 rx=0x7f63f405c070 tx=0x7f63f400e670).stop 2021-03-09T09:58:58.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:58.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:58:58.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:58:58.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:58:58.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:58:58.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:58.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:58:58.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:58.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:58:58.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:58:58.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:58:58.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:58.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064f50 con 0x7f63d8003400 2021-03-09T09:58:58.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:58.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3a0 con 0x7f63d8008840 2021-03-09T09:58:58.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:58:58.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e7f0 con 0x7f6400146050 2021-03-09T09:58:58.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:58:58.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:58:58.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:58.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:58:58.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.085+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:58:58.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.086+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:58:58.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.086+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:58:58.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.086+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:58:58.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.086+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:58:58.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.086+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:58:58.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.086+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:58:58.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.086+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:58:58.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:58:58.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:58.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:58.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:58:58.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:58:58.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:58:58.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:58:58.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:58.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:58.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:58:58.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:58:58.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:58:58.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:58:58.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:58:58.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.087+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:58.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.088+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11639 cs=0 l=1 rev1=1 rx=0x7f63f404d040 tx=0x7f63f4053ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:58:58.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.088+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:58:58.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.088+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:58:58.100 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:58:58.089+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32000 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:58:58.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.090+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f6400146050 2021-03-09T09:58:58.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.090+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:58:58.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.090+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:58:58.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.090+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:58:58.102 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:58:58.102 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:58:58.102 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:58.103 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:58:58.103 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:58:58.103 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:58:58.103 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:58:58.104 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:58:58.104 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:58:58.104 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:58:58.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.091+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T09:58:58.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.091+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:58:58.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.091+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:58:58.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40532c0 con 0x7f6400146050 2021-03-09T09:58:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:58:58.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:58.628+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:58:59.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:59.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:58:59.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:58:59.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:00.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:00.159+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:00.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:00.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:00.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:00.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:01.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:01.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:01.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:01.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:02.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:02.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:02.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:02.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:03.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:03.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:03.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11639 cs=0 l=1 rev1=1 rx=0x7f63f404d040 tx=0x7f63f4053ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:03.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11639 cs=0 l=1 rev1=1 rx=0x7f63f404d040 tx=0x7f63f4053ee0).stop 2021-03-09T09:59:03.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:03.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:03.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:03.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:03.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:59:03.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:03.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:03.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:03.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:03.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:03.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:03.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:03.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:59:03.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:03.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cf70 con 0x7f63d8008840 2021-03-09T09:59:03.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:03.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3a0 con 0x7f63e0018390 2021-03-09T09:59:03.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:03.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:03.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:03.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.090+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:03.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.090+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:03.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.092+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:03.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.092+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:59:03.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.092+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:03.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.092+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:03.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.092+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:03.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.092+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:03.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.092+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:03.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:03.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:03.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:03.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:03.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:03.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:59:03.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:59:03.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:03.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:03.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:59:03.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:03.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:03.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ccf0 con 0x7f63e0018390 2021-03-09T09:59:03.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:59:03.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:03.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:59:03.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:03.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.093+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11655 cs=0 l=1 rev1=1 rx=0x7f63fc078dc0 tx=0x7f63fc0039e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:03.106 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:03.094+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32038 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:03.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f63e0018390 2021-03-09T09:59:03.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.095+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:03.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.095+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:03.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.095+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:03.108 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:03.108 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:03.109 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:03.109 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:03.109 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:03.109 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:03.109 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:03.110 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:03.110 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:03.110 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:03.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0333f0 con 0x7f63e0018390 2021-03-09T09:59:03.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.098+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:03.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.098+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:59:03.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.099+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T09:59:03.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:03.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:03.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:04.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:04.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:04.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:04.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:05.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:05.160+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:05.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:05.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:05.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:05.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:06.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:06.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:06.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:06.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:07.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:07.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:07.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:08.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:08.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:08.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11655 cs=0 l=1 rev1=1 rx=0x7f63fc078dc0 tx=0x7f63fc0039e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:08.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11655 cs=0 l=1 rev1=1 rx=0x7f63fc078dc0 tx=0x7f63fc0039e0).stop 2021-03-09T09:59:08.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:08.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:08.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:08.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:08.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:59:08.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:08.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:08.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:08.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:08.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:08.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:08.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:08.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00134e0 con 0x7f63d8003400 2021-03-09T09:59:08.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:08.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63d8008840 2021-03-09T09:59:08.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:08.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T09:59:08.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:08.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:08.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:08.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:08.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:08.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.096+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:08.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.096+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:59:08.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.096+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:08.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.096+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:08.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.096+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:08.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.096+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:08.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.096+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:08.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:08.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:08.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:08.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:08.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:08.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:59:08.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:59:08.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:08.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:08.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022a00 con 0x7f6400146050 2021-03-09T09:59:08.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:08.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:08.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:59:08.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:59:08.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:08.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:59:08.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11664 cs=0 l=1 rev1=1 rx=0x7f63e00296b0 tx=0x7f63fc033460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:08.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.097+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:08.110 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:08.100+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32078 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:08.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065c10 con 0x7f6400146050 2021-03-09T09:59:08.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.100+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:08.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.100+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:08.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.100+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:08.112 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:08.112 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:08.113 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:08.113 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:08.113 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:08.113 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:08.113 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:08.114 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:08.114 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:08.114 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:08.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f6400146050 2021-03-09T09:59:08.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.100+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:08.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.100+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:59:08.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04b9c0 con 0x7f6400146050 2021-03-09T09:59:08.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:08.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:08.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:08.630+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:09.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:09.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:09.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:09.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:10.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:10.161+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:10.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:10.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:10.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:10.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:11.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:11.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:11.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:11.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:12.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:12.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:12.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:13.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:13.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:13.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11664 cs=0 l=1 rev1=1 rx=0x7f63e00296b0 tx=0x7f63fc033460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:13.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11664 cs=0 l=1 rev1=1 rx=0x7f63e00296b0 tx=0x7f63fc033460).stop 2021-03-09T09:59:13.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:13.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:13.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:13.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:13.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:59:13.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:13.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:13.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:13.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:13.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:13.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:13.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:13.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027e50 con 0x7f63d8003400 2021-03-09T09:59:13.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:13.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T09:59:13.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:13.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005edf0 con 0x7f63e0018390 2021-03-09T09:59:13.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:13.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:13.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:13.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:13.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.101+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:13.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.102+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:13.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.102+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:59:13.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.102+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:13.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.102+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:13.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:13.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:13.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:13.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:13.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:13.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:13.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:13.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:13.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:59:13.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:59:13.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:13.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:13.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005de90 con 0x7f63e0018390 2021-03-09T09:59:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022a00 con 0x7f63e0018390 2021-03-09T09:59:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:59:13.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.103+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:13.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.104+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11679 cs=0 l=1 rev1=1 rx=0x7f63e0029680 tx=0x7f63fc066d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:13.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.104+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:59:13.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.104+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:13.117 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:13.105+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32116 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:13.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f63e0018390 2021-03-09T09:59:13.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.106+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:13.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.106+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:13.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.106+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:13.119 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:13.119 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:13.119 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:13.119 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:13.119 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:13.120 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:13.120 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:13.120 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:13.120 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:13.120 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.107+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f63e0018390 2021-03-09T09:59:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.107+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.107+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:59:13.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04b9c0 con 0x7f63e0018390 2021-03-09T09:59:13.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:13.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:13.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:14.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:14.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:14.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:14.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:15.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:15.162+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:15.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:15.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:15.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:15.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:16.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:16.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:16.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:16.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:17.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:17.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:17.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:17.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:18.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:18.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:18.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11679 cs=0 l=1 rev1=1 rx=0x7f63e0029680 tx=0x7f63fc066d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:18.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11679 cs=0 l=1 rev1=1 rx=0x7f63e0029680 tx=0x7f63fc066d60).stop 2021-03-09T09:59:18.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:18.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:18.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:18.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:18.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T09:59:18.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:18.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:18.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:18.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:18.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:18.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:18.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:18.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e390 con 0x7f63d8003400 2021-03-09T09:59:18.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:18.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T09:59:18.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:18.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3a0 con 0x7f6400146050 2021-03-09T09:59:18.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:18.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:18.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:18.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:18.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.106+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:18.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.107+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:18.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.107+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:59:18.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.107+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:18.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.107+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:18.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.107+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:18.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.107+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:18.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.107+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:18.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:18.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:18.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:18.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:18.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:18.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:59:18.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:59:18.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:18.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:18.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:59:18.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:18.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:18.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:59:18.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:59:18.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:18.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11694 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4005230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:18.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:59:18.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.109+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:18.123 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:18.111+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32156 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:18.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.111+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f406a2d0 con 0x7f6400146050 2021-03-09T09:59:18.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.111+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:18.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.111+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:18.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.111+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:18.124 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:18.125 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:18.125 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:18.125 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:18.125 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:18.125 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:18.126 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:18.126 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:18.126 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:18.126 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:18.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.113+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4073360 con 0x7f6400146050 2021-03-09T09:59:18.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.113+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:18.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.114+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:59:18.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.114+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f406a2d0 con 0x7f6400146050 2021-03-09T09:59:18.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:18.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:18.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:18.631+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:19.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:19.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:19.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:19.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:20.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:20.163+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:20.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:20.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:20.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:20.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:21.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:21.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:21.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:21.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:22.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:22.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:22.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:22.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:23.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:23.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:23.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11694 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4005230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:23.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11694 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4005230).stop 2021-03-09T09:59:23.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:23.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:23.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:23.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:23.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:59:23.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:23.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:23.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:23.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:23.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:23.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:23.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:23.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:59:23.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:23.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc00 con 0x7f63d8008840 2021-03-09T09:59:23.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:23.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065cc0 con 0x7f63e0018390 2021-03-09T09:59:23.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:23.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:23.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:23.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:23.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:23.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.112+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:23.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.112+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:59:23.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.112+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:23.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.112+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:23.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.112+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.112+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.112+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:59:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:59:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022a00 con 0x7f63e0018390 2021-03-09T09:59:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:59:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:59:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.113+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.114+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:59:23.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.114+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:23.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.114+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11706 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f401bf00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:23.127 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:23.116+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32194 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:23.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401ae70 con 0x7f63e0018390 2021-03-09T09:59:23.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.117+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.117+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.117+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:23.131 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.117+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f401ae70 con 0x7f63e0018390 2021-03-09T09:59:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.117+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:23.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.117+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:59:23.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405bb40 con 0x7f63e0018390 2021-03-09T09:59:23.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:23.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:23.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:24.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:24.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:24.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:24.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:25.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:25.164+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:25.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:25.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:25.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:25.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:26.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:26.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:26.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:26.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:27.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:27.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:27.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:27.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:28.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:28.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:28.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11706 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f401bf00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:28.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11706 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f401bf00).stop 2021-03-09T09:59:28.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:28.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:28.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:28.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:28.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T09:59:28.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:28.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:28.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:28.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:28.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:28.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:28.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:28.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e7f0 con 0x7f63d8003400 2021-03-09T09:59:28.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:28.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3a0 con 0x7f63d8008840 2021-03-09T09:59:28.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:28.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e390 con 0x7f6400146050 2021-03-09T09:59:28.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:28.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:28.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.116+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:28.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:28.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:28.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:28.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:59:28.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:28.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:28.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:28.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:28.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:28.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:28.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:28.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:28.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:28.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:28.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:59:28.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:59:28.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:28.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:28.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:59:28.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:28.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:28.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T09:59:28.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.118+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:59:28.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.118+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:28.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.118+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:59:28.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.118+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:28.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.118+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11717 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc0715c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:28.132 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:28.120+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32234 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:28.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.122+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066dd0 con 0x7f6400146050 2021-03-09T09:59:28.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.122+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:28.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.122+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:28.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.122+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:28.134 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:28.134 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:28.136 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:28.136 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:28.136 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:28.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.122+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T09:59:28.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.122+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:28.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.122+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:59:28.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.122+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T09:59:28.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:28.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:28.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:28.633+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:29.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:29.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:29.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:29.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:30.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:30.165+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:30.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:30.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:30.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:30.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:31.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:31.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:31.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:31.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:32.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:32.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:32.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:32.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:33.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:33.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:33.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11717 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc0715c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:33.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11717 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc0715c0).stop 2021-03-09T09:59:33.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:33.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:33.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:33.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:33.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:59:33.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:33.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:33.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:33.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:33.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:33.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:33.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:33.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:59:33.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:33.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b8f0 con 0x7f63d8008840 2021-03-09T09:59:33.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:33.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b740 con 0x7f63e0018390 2021-03-09T09:59:33.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:33.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:33.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.121+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:33.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.122+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:33.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.122+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:33.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.122+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:33.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.122+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:59:33.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.122+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:33.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.122+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:33.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:33.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:33.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:33.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:33.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:33.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:33.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:33.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:33.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:59:33.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:59:33.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:33.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:33.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:59:33.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:33.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:33.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022a00 con 0x7f63e0018390 2021-03-09T09:59:33.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:59:33.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.123+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:33.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.124+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11735 cs=0 l=1 rev1=1 rx=0x7f63f4062480 tx=0x7f63f404cfa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:33.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.124+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:59:33.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.124+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:33.137 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:33.125+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32270 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:33.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034390 con 0x7f63e0018390 2021-03-09T09:59:33.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.127+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:33.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.127+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:33.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.127+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:33.139 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:33.139 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:33.139 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:33.139 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:33.140 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:33.140 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:33.140 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:33.141 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40341b0 con 0x7f63e0018390 2021-03-09T09:59:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.127+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.127+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:59:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f406a2d0 con 0x7f63e0018390 2021-03-09T09:59:33.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:33.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:33.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:34.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:34.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:34.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:34.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:35.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:35.166+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:35.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:35.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:35.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:35.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:36.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:36.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:36.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:37.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:37.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:37.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:37.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:38.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T09:59:38.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:38.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11735 cs=0 l=1 rev1=1 rx=0x7f63f4062480 tx=0x7f63f404cfa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:38.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11735 cs=0 l=1 rev1=1 rx=0x7f63f4062480 tx=0x7f63f404cfa0).stop 2021-03-09T09:59:38.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:38.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:38.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:38.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:38.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:59:38.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:38.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:38.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:38.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:38.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:38.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:38.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:38.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017290 con 0x7f63d8003400 2021-03-09T09:59:38.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:38.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ea30 con 0x7f63d8008840 2021-03-09T09:59:38.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:38.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T09:59:38.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:38.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:38.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:38.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:38.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:38.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.126+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:38.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.127+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:59:38.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.127+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:38.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.127+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:38.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.128+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:38.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.128+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:38.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.128+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:38.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:38.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:38.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:38.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:38.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:38.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:59:38.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:59:38.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:38.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:38.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a700 con 0x7f6400146050 2021-03-09T09:59:38.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:38.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:38.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T09:59:38.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:59:38.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.129+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:38.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.130+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:59:38.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.130+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11752 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4034290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:38.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.130+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:38.143 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:38.131+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32312 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:38.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f6400146050 2021-03-09T09:59:38.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.131+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:38.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.132+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:38.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.132+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:38.145 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:38.145 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:38.145 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:38.146 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:38.146 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:38.146 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:38.146 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:38.147 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:38.147 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:38.147 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:38.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f6400146050 2021-03-09T09:59:38.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.135+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:38.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f404cbc0 con 0x7f6400146050 2021-03-09T09:59:38.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.135+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:59:38.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:38.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:38.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:38.634+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:39.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:39.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:39.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:39.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:40.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:40.167+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:40.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:40.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:40.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:40.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:41.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:41.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:42.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:42.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:42.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:42.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11752 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4034290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:43.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11752 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4034290).stop 2021-03-09T09:59:43.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:43.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:43.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:43.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:43.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T09:59:43.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:43.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:43.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:43.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:43.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:43.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:43.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:43.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f6b0 con 0x7f63d8003400 2021-03-09T09:59:43.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:43.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b950 con 0x7f63d8008840 2021-03-09T09:59:43.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:43.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T09:59:43.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:43.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:43.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.131+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:43.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:43.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:43.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:43.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:59:43.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:43.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:43.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:43.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:43.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:43.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:43.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:43.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:43.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:43.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:43.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T09:59:43.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T09:59:43.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:43.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:43.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T09:59:43.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:43.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:43.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016190 con 0x7f63e0018390 2021-03-09T09:59:43.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T09:59:43.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.132+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:43.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.133+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11760 cs=0 l=1 rev1=1 rx=0x7f63e00121c0 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:43.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.133+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:59:43.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.133+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:43.147 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:43.136+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32350 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:43.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T09:59:43.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.137+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:43.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.137+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:43.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.137+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:43.149 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:43.149 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:43.149 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:43.149 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:43.150 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:43.150 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:43.150 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:43.150 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:43.151 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:43.151 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:43.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00541f0 con 0x7f63e0018390 2021-03-09T09:59:43.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.137+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:43.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.137+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:59:43.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028310 con 0x7f63e0018390 2021-03-09T09:59:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:43.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:44.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:44.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:44.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:44.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:45.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:45.168+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:45.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:45.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:45.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:46.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:46.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:46.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:46.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:47.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:47.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:47.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:47.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:48.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:48.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:48.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11760 cs=0 l=1 rev1=1 rx=0x7f63e00121c0 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:48.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11760 cs=0 l=1 rev1=1 rx=0x7f63e00121c0 tx=0x7f63f0000c30).stop 2021-03-09T09:59:48.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:48.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:48.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:48.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11760 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:48.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:59:48.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:48.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:48.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:48.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:48.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:48.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:48.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:48.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002d80 con 0x7f63d8003400 2021-03-09T09:59:48.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:48.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T09:59:48.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:48.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011df0 con 0x7f6400146050 2021-03-09T09:59:48.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:48.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:48.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:48.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:48.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:48.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:59:48.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:48.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:48.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:48.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:48.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:48.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:48.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:48.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:48.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:48.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:48.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:48.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:59:48.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:59:48.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:48.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:48.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T09:59:48.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:48.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:48.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a700 con 0x7f6400146050 2021-03-09T09:59:48.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:59:48.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.138+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:48.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.139+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11773 cs=0 l=1 rev1=1 rx=0x7f63f402c880 tx=0x7f63f40632c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:48.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.139+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:59:48.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.139+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:48.164 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:48.142+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32399 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:48.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400d1c0 con 0x7f6400146050 2021-03-09T09:59:48.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.143+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:48.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.143+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:48.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.143+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:48.166 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:48.166 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:48.166 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:48.167 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:48.167 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:48.167 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:48.168 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:48.168 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:48.168 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:48.168 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:48.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400d1c0 con 0x7f6400146050 2021-03-09T09:59:48.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.147+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:48.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.147+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:59:48.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4073360 con 0x7f6400146050 2021-03-09T09:59:48.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:48.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:48.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:48.635+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:49.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:49.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:49.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:49.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:50.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:50.169+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:50.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:50.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:50.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:50.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:51.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:51.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:51.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:51.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:52.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:52.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:52.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:52.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:53.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:53.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:53.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11773 cs=0 l=1 rev1=1 rx=0x7f63f402c880 tx=0x7f63f40632c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:53.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11773 cs=0 l=1 rev1=1 rx=0x7f63f402c880 tx=0x7f63f40632c0).stop 2021-03-09T09:59:53.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:53.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:53.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:53.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:53.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T09:59:53.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:53.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:53.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:53.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:53.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:53.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:53.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:53.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T09:59:53.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:53.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005edf0 con 0x7f63d8008840 2021-03-09T09:59:53.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:53.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266d0 con 0x7f63e0018390 2021-03-09T09:59:53.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:53.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:53.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:53.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:53.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:53.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.143+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:53.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T09:59:53.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:53.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:53.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:53.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:53.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:53.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:53.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:53.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:53.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:53.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:53.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T09:59:53.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T09:59:53.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:53.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:53.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T09:59:53.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:53.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:53.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T09:59:53.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T09:59:53.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.144+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:53.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.145+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11791 cs=0 l=1 rev1=1 rx=0x7f63f402c620 tx=0x7f63f400d230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:53.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.145+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T09:59:53.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.145+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:53.159 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:53.148+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32437 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:53.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402c8f0 con 0x7f63e0018390 2021-03-09T09:59:53.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.149+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:53.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.149+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:53.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.149+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:53.161 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:53.161 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:53.161 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:53.161 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:53.162 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:53.162 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:53.162 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:53.162 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:53.163 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:53.163 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:53.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.152+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0018390 2021-03-09T09:59:53.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.152+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:53.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.152+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T09:59:53.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.152+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405c2f0 con 0x7f63e0018390 2021-03-09T09:59:53.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:53.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:53.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:54.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:54.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:54.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:54.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:55.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:55.170+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T09:59:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:55.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:55.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:56.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:56.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:56.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:56.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:57.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:57.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:57.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:57.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:58.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T09:59:58.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T09:59:58.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11791 cs=0 l=1 rev1=1 rx=0x7f63f402c620 tx=0x7f63f400d230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T09:59:58.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11791 cs=0 l=1 rev1=1 rx=0x7f63f402c620 tx=0x7f63f400d230).stop 2021-03-09T09:59:58.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:58.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T09:59:58.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T09:59:58.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T09:59:58.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T09:59:58.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:58.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T09:59:58.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:58.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T09:59:58.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T09:59:58.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T09:59:58.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:58.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065cc0 con 0x7f63d8003400 2021-03-09T09:59:58.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:58.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011df0 con 0x7f63d8008840 2021-03-09T09:59:58.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T09:59:58.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400146050 2021-03-09T09:59:58.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T09:59:58.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T09:59:58.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:58.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T09:59:58.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T09:59:58.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T09:59:58.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T09:59:58.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T09:59:58.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T09:59:58.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T09:59:58.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T09:59:58.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.149+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T09:59:58.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T09:59:58.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:58.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:58.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T09:59:58.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T09:59:58.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T09:59:58.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T09:59:58.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:58.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:58.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T09:59:58.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T09:59:58.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T09:59:58.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f6400146050 2021-03-09T09:59:58.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T09:59:58.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.150+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:58.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.152+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11805 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T09:59:58.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.152+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T09:59:58.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.152+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T09:59:58.164 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T09:59:58.154+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32477 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T09:59:58.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013cf0 con 0x7f6400146050 2021-03-09T09:59:58.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.156+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T09:59:58.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.156+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T09:59:58.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.156+0000 7f63edffb700 10 monclient: dump: 2021-03-09T09:59:58.166 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T09:59:58.166 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T09:59:58.166 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:58.166 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T09:59:58.166 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T09:59:58.167 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T09:59:58.167 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T09:59:58.167 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T09:59:58.167 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T09:59:58.168 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T09:59:58.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T09:59:58.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.156+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T09:59:58.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.156+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T09:59:58.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T09:59:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:58.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T09:59:58.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:58.637+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T09:59:59.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:59.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T09:59:59.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T09:59:59.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:00.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:00.171+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:00.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:00.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:01.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:01.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:01.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:01.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:02.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:02.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:02.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:02.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:03.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:03.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:03.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11805 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:03.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11805 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc009a70).stop 2021-03-09T10:00:03.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:03.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:03.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:03.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:03.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:00:03.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:03.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:03.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:03.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:03.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:03.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:03.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:03.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022c50 con 0x7f63d8003400 2021-03-09T10:00:03.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:03.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:00:03.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:03.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066ab0 con 0x7f63e0018390 2021-03-09T10:00:03.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:03.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:03.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:03.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:03.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:03.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:03.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:00:03.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:03.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:03.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:03.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:03.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.155+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:03.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:03.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:03.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:03.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:03.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:03.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:00:03.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:00:03.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:03.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:03.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012d90 con 0x7f63e0018390 2021-03-09T10:00:03.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:03.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:00:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:00:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.156+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:03.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.157+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11816 cs=0 l=1 rev1=1 rx=0x7f63f0038fd0 tx=0x7f63f0030300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:03.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.157+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:00:03.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.157+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:03.170 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:03.159+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32515 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:03.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.160+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63e0018390 2021-03-09T10:00:03.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.160+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:03.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.160+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:03.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.160+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:03.172 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:03.172 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:03.173 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:03.173 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:03.173 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:03.173 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:03.173 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:03.174 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:03.174 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:03.174 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:03.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63e0018390 2021-03-09T10:00:03.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.163+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:03.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.163+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:00:03.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:00:03.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:03.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:03.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:04.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:04.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:04.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:04.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:05.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:05.172+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:05.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:05.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:05.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:05.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:06.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:06.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:06.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:06.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:07.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:07.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:07.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:07.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:08.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:08.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:08.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11816 cs=0 l=1 rev1=1 rx=0x7f63f0038fd0 tx=0x7f63f0030300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:08.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11816 cs=0 l=1 rev1=1 rx=0x7f63f0038fd0 tx=0x7f63f0030300).stop 2021-03-09T10:00:08.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:08.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:08.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:08.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:08.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:00:08.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:08.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:08.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:08.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:08.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:08.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:08.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:08.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8003400 2021-03-09T10:00:08.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:08.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T10:00:08.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:08.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065cc0 con 0x7f6400146050 2021-03-09T10:00:08.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:08.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:08.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:08.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:08.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:08.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:08.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:00:08.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:08.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:08.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:08.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:08.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:08.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:08.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:08.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:08.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:08.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:08.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:00:08.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:00:08.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:08.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:08.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T10:00:08.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:08.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:08.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012d90 con 0x7f6400146050 2021-03-09T10:00:08.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:00:08.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:08.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:00:08.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:08.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.161+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11830 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f005eb80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:08.178 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:08.165+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32555 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:08.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005ee80 con 0x7f6400146050 2021-03-09T10:00:08.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.165+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:08.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.165+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:08.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.165+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:08.180 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:08.181 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:08.181 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:08.181 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:08.182 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:08.182 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:08.182 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:08.182 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:08.182 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:08.183 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:08.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005ee80 con 0x7f6400146050 2021-03-09T10:00:08.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.166+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:08.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.166+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:00:08.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e460 con 0x7f6400146050 2021-03-09T10:00:08.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:08.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:08.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:08.638+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:09.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:09.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:09.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:09.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:10.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:10.173+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:10.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:10.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:10.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:10.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:11.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:11.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:11.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:11.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:12.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:12.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:12.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:12.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:13.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:13.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:13.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11830 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f005eb80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:13.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11830 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f005eb80).stop 2021-03-09T10:00:13.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:13.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:13.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:13.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:13.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:00:13.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:13.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:13.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:13.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:13.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:13.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:13.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:13.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cf90 con 0x7f63d8003400 2021-03-09T10:00:13.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:13.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:00:13.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:13.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000eb50 con 0x7f63e0018390 2021-03-09T10:00:13.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:13.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:13.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:13.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:13.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:13.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:13.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6406553700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T10:00:13.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:00:13.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:13.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:13.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2021-03-09T10:00:13.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2021-03-09T10:00:13.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:13.177 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:13.171+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32593 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:13.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:13.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T10:00:13.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:13.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T10:00:13.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:13.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:00:13.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:13.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:13.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:13.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:13.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:13.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:13.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:13.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:13.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:13.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:00:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:00:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079570 con 0x7f63e0018390 2021-03-09T10:00:13.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:13.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:13.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005de90 con 0x7f63e0018390 2021-03-09T10:00:13.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:00:13.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:13.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:00:13.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11855 cs=0 l=1 rev1=1 rx=0x7f63e001c6e0 tx=0x7f63fc052e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:13.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:13.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63e0018390 2021-03-09T10:00:13.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:13.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:13.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:13.378 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:13.378 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:13.378 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:13.379 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:13.379 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:13.379 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:13.379 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:13.380 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:13.380 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:13.380 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:13.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc033340 con 0x7f63e0018390 2021-03-09T10:00:13.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:13.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.367+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:00:13.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:00:13.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:13.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:13.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:14.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:14.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:14.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:14.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:15.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:15.174+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:15.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:15.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:15.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:15.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:16.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:16.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:16.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:16.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:17.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:17.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:17.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:17.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:18.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:18.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:18.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11855 cs=0 l=1 rev1=1 rx=0x7f63e001c6e0 tx=0x7f63fc052e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:18.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11855 cs=0 l=1 rev1=1 rx=0x7f63e001c6e0 tx=0x7f63fc052e60).stop 2021-03-09T10:00:18.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:18.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:18.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:18.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:18.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:00:18.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:18.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:18.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:18.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:18.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:18.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:18.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:18.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c0f0 con 0x7f63d8003400 2021-03-09T10:00:18.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:18.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ade0 con 0x7f63d8008840 2021-03-09T10:00:18.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:18.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065cc0 con 0x7f6400146050 2021-03-09T10:00:18.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:18.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:18.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:18.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.173+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:18.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.173+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:18.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:18.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:00:18.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:18.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:18.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:18.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:18.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:18.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:18.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:18.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:18.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:18.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:18.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:00:18.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:00:18.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:18.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:18.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T10:00:18.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:18.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:18.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T10:00:18.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:00:18.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.174+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:18.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.175+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:00:18.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.175+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11862 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f40701e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:18.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.175+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:18.187 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:18.176+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32633 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:18.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f6400146050 2021-03-09T10:00:18.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.178+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:18.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.178+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:18.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.178+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:18.189 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:18.189 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:18.189 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:18.190 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:18.190 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:18.190 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:18.190 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:18.190 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:18.191 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:18.191 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:18.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f6400146050 2021-03-09T10:00:18.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.179+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:18.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.179+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:00:18.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.180+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f401ac40 con 0x7f6400146050 2021-03-09T10:00:18.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:18.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:18.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:18.639+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:19.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:19.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:19.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:19.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:20.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:20.175+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:20.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:20.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:20.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:20.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:21.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:21.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:21.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:21.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:22.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:22.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:22.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:22.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:23.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:23.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:23.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11862 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f40701e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:23.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11862 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f40701e0).stop 2021-03-09T10:00:23.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:23.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:23.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:23.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:23.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:00:23.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:23.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:23.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002baf0 con 0x7f63d8003400 2021-03-09T10:00:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:23.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022c50 con 0x7f63d8008840 2021-03-09T10:00:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:00:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:23.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.177+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:23.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.178+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:23.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.178+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:00:23.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.178+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:23.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.178+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:23.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.178+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:23.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.178+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:23.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.178+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:23.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:23.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:23.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:23.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:23.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:23.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:00:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:00:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:00:23.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079570 con 0x7f63e0018390 2021-03-09T10:00:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:00:23.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.179+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:23.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.180+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11874 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04bd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:23.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.180+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:00:23.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.180+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:23.196 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:23.182+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32667 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:23.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.182+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06d540 con 0x7f63e0018390 2021-03-09T10:00:23.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.182+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:23.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.183+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:23.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.183+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:23.198 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:23.198 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:23.198 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:23.199 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:23.199 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:23.199 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:23.200 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:23.200 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:23.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.184+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06d540 con 0x7f63e0018390 2021-03-09T10:00:23.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.184+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:23.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.184+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:00:23.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06d170 con 0x7f63e0018390 2021-03-09T10:00:23.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:23.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:23.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:24.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:24.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:24.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:24.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:25.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:25.176+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:25.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:25.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:25.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:25.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:26.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:26.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:26.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:26.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:27.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:27.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:27.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:27.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:28.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:28.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:28.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11874 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04bd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:28.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11874 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04bd90).stop 2021-03-09T10:00:28.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:28.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:28.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:28.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:28.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:00:28.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:28.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:28.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:28.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:28.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:28.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:28.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:28.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000eb50 con 0x7f63d8003400 2021-03-09T10:00:28.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:28.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T10:00:28.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:28.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3a0 con 0x7f6400146050 2021-03-09T10:00:28.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:28.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:28.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.183+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:28.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.184+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:28.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.184+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:28.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.184+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:28.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.184+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:00:28.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.184+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:28.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.184+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:28.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.185+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:28.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.185+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:28.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.185+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:28.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:28.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:28.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:28.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:28.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:28.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:00:28.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:00:28.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:28.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:28.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001add0 con 0x7f6400146050 2021-03-09T10:00:28.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:28.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:28.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T10:00:28.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:00:28.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.186+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:28.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.187+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11894 cs=0 l=1 rev1=1 rx=0x7f63fc05f010 tx=0x7f63fc06fb30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:28.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.187+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:00:28.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.187+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:28.201 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:28.188+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32711 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:28.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc008c90 con 0x7f6400146050 2021-03-09T10:00:28.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.188+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:28.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.188+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:28.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.188+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:28.203 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:28.203 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:28.203 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:28.204 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:28.204 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:28.204 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:28.204 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:28.205 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:28.205 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:28.205 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:28.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc008c90 con 0x7f6400146050 2021-03-09T10:00:28.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.190+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:28.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.190+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:00:28.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f6400146050 2021-03-09T10:00:28.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:28.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:28.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:28.641+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:29.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:29.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:29.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:29.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:30.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:30.177+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:30.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:30.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:30.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:30.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:31.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:31.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:31.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:31.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:32.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:32.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:32.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:32.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:33.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:33.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:33.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11894 cs=0 l=1 rev1=1 rx=0x7f63fc05f010 tx=0x7f63fc06fb30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:33.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11894 cs=0 l=1 rev1=1 rx=0x7f63fc05f010 tx=0x7f63fc06fb30).stop 2021-03-09T10:00:33.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:33.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:33.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:33.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:33.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:00:33.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:33.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.189+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:33.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:33.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:33.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:33.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:33.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:33.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065cc0 con 0x7f63d8003400 2021-03-09T10:00:33.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:33.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00136e0 con 0x7f63d8008840 2021-03-09T10:00:33.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:33.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:00:33.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:33.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:33.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:33.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:33.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.190+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:33.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:33.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:00:33.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:33.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:33.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:33.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:33.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:33.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:33.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:33.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:33.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:33.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:33.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:00:33.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.191+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:00:33.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:33.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:33.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005eda0 con 0x7f63e0018390 2021-03-09T10:00:33.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:33.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:33.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T10:00:33.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:00:33.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:33.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:00:33.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:33.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.192+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11902 cs=0 l=1 rev1=1 rx=0x7f63f003b280 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:33.205 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:33.194+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32749 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:33.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:00:33.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.195+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:33.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.195+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:33.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.195+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:33.207 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:33.208 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:33.208 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:33.208 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:33.208 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:33.209 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:33.209 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:33.209 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:33.209 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:33.209 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:33.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.197+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:00:33.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.197+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:33.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.197+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:00:33.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005ed50 con 0x7f63e0018390 2021-03-09T10:00:33.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:33.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:33.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:34.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:34.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:34.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:34.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:35.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:35.178+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:35.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:35.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:35.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:35.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:36.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:36.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:36.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:36.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:37.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:37.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:37.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:37.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:38.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.194+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:38.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.194+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:38.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11902 cs=0 l=1 rev1=1 rx=0x7f63f003b280 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:38.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11902 cs=0 l=1 rev1=1 rx=0x7f63f003b280 tx=0x7f63f0006610).stop 2021-03-09T10:00:38.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.194+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:38.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.194+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:38.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:38.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11902 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:38.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.194+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:00:38.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:38.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:38.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:38.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:38.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:38.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:38.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:38.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024260 con 0x7f63d8003400 2021-03-09T10:00:38.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:38.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de80 con 0x7f63d8008840 2021-03-09T10:00:38.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:38.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065cc0 con 0x7f6400146050 2021-03-09T10:00:38.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:38.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:38.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:38.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:38.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:38.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:38.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:00:38.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:38.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:38.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:38.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:38.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.195+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:38.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:38.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:38.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:38.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:38.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:38.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:00:38.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:00:38.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:38.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:38.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T10:00:38.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:38.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:38.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005eda0 con 0x7f6400146050 2021-03-09T10:00:38.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:00:38.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.196+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:38.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11914 cs=0 l=1 rev1=1 rx=0x7f63e006b850 tx=0x7f63f0006c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:38.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.198+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:00:38.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.198+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:38.210 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:38.199+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32787 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:38.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007ec70 con 0x7f6400146050 2021-03-09T10:00:38.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.200+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:38.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.200+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:38.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.200+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:38.211 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:38.212 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:38.212 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:38.212 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:38.212 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:38.213 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:38.213 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:38.213 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:38.213 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:38.213 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:38.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.203+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003b480 con 0x7f6400146050 2021-03-09T10:00:38.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.203+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:38.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.203+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:00:38.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.204+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ec70 con 0x7f6400146050 2021-03-09T10:00:38.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:38.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:38.642+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:39.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:39.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:39.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:39.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:40.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:40.179+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:40.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:40.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:40.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:41.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:41.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:42.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:42.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:42.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:42.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:43.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:43.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:43.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11914 cs=0 l=1 rev1=1 rx=0x7f63e006b850 tx=0x7f63f0006c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:43.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11914 cs=0 l=1 rev1=1 rx=0x7f63e006b850 tx=0x7f63f0006c30).stop 2021-03-09T10:00:43.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:43.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:43.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:43.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:43.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:00:43.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:43.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:43.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:43.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:43.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:43.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:43.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:43.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024ac0 con 0x7f63d8003400 2021-03-09T10:00:43.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:43.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d760 con 0x7f63d8008840 2021-03-09T10:00:43.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:43.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63e0018390 2021-03-09T10:00:43.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:43.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:43.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.200+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:43.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.201+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:43.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.201+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:43.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:43.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:00:43.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:43.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:43.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:43.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:43.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:43.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:43.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:43.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:43.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:43.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:43.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:00:43.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.202+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:00:43.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:43.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:43.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:00:43.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:43.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:43.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:00:43.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:00:43.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:43.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11931 cs=0 l=1 rev1=1 rx=0x7f63f0038c80 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:43.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:00:43.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.203+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:43.215 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:43.205+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32825 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:43.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005ebf0 con 0x7f63e0018390 2021-03-09T10:00:43.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.207+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:43.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.207+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:43.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.207+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:43.217 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:43.217 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:43.217 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:43.217 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:43.217 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:43.218 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:43.218 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:43.218 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:43.218 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:43.219 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:43.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041da0 con 0x7f63e0018390 2021-03-09T10:00:43.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.207+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:43.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.207+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:00:43.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.210+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036890 con 0x7f63e0018390 2021-03-09T10:00:43.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:43.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:43.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:44.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:44.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:44.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:44.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:45.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:45.180+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:45.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:45.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:45.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:45.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:46.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:46.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:46.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:46.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:47.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:47.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:47.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:47.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:48.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.205+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:00:48.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.205+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:48.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.205+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11931 cs=0 l=1 rev1=1 rx=0x7f63f0038c80 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:48.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.205+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11931 cs=0 l=1 rev1=1 rx=0x7f63f0038c80 tx=0x7f63f0000c30).stop 2021-03-09T10:00:48.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.205+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:48.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:48.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:48.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:48.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:00:48.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:48.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:48.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:48.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:48.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:48.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:48.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:48.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:00:48.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:48.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065cc0 con 0x7f63d8008840 2021-03-09T10:00:48.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:48.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005edf0 con 0x7f6400146050 2021-03-09T10:00:48.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:48.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:48.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:48.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:48.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:00:48.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:48.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.206+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:48.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.207+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:48.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.207+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:48.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.207+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:48.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.207+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:48.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.207+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:48.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:48.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:48.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:48.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:48.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:48.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:00:48.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:00:48.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:48.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:48.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T10:00:48.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:48.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:48.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001add0 con 0x7f6400146050 2021-03-09T10:00:48.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:00:48.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:48.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11940 cs=0 l=1 rev1=1 rx=0x7f63fc047e30 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:48.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:00:48.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.208+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:48.222 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:48.210+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32866 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:48.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f6400146050 2021-03-09T10:00:48.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.211+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:48.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.211+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:48.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.211+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:48.224 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:48.224 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:48.224 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:48.224 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:48.225 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:48.226 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:48.226 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:48.227 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:48.227 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:48.227 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:48.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T10:00:48.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.211+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:48.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.211+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:00:48.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.212+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T10:00:48.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:48.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:48.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:48.643+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:49.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:49.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:49.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:49.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:50.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:50.181+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:50.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:50.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:50.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:50.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:51.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:51.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:51.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:51.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:52.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:52.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:52.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:52.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:53.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:53.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:53.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11940 cs=0 l=1 rev1=1 rx=0x7f63fc047e30 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:53.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11940 cs=0 l=1 rev1=1 rx=0x7f63fc047e30 tx=0x7f63fc003a50).stop 2021-03-09T10:00:53.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:53.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:53.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:53.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:53.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:00:53.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:53.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:53.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:53.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:53.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:53.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:53.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:53.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00153c0 con 0x7f63d8003400 2021-03-09T10:00:53.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:53.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068b50 con 0x7f63d8008840 2021-03-09T10:00:53.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:53.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0018390 2021-03-09T10:00:53.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:53.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:53.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.210+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:53.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:53.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:53.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:53.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:00:53.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:53.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:53.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:53.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:53.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:53.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.211+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:00:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:00:53.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:53.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:53.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024c30 con 0x7f63e0018390 2021-03-09T10:00:53.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:00:53.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:00:53.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:53.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11953 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0028310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:53.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:00:53.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.212+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:53.228 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:53.216+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32903 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:00:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.216+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.216+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:53.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.216+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:53.230 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:53.230 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:53.230 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:53.231 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:53.232 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:53.232 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:53.232 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:53.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:00:53.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.216+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:53.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.216+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:00:53.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.217+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042320 con 0x7f63e0018390 2021-03-09T10:00:53.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:53.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:53.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:54.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:54.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:54.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:55.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:55.181+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:00:55.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:55.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:55.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:55.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:56.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:56.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:56.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:56.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:57.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:57.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:57.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:57.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:58.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:00:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:00:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11953 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0028310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:00:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11953 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0028310).stop 2021-03-09T10:00:58.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:58.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:00:58.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:00:58.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:00:58.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:00:58.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:58.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:00:58.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:58.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:00:58.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:00:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:00:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017b50 con 0x7f63d8003400 2021-03-09T10:00:58.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b6a0 con 0x7f63d8008840 2021-03-09T10:00:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:00:58.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065cc0 con 0x7f6400146050 2021-03-09T10:00:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.216+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:00:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:00:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:58.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:00:58.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:00:58.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:00:58.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:00:58.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:00:58.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:00:58.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:00:58.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:00:58.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.217+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:00:58.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:00:58.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:58.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:58.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:00:58.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:00:58.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:00:58.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:00:58.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:58.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:58.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T10:00:58.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:00:58.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:00:58.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T10:00:58.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:00:58.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.218+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:58.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.219+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11967 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4005180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:00:58.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.219+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:00:58.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.219+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:00:58.233 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:00:58.222+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32933 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:00:58.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401ac40 con 0x7f6400146050 2021-03-09T10:00:58.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.222+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:00:58.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.222+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:00:58.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.222+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:00:58.235 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:00:58.235 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:00:58.235 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:58.235 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:00:58.236 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:00:58.236 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:00:58.236 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:00:58.236 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:00:58.236 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:00:58.237 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:00:58.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4006a90 con 0x7f6400146050 2021-03-09T10:00:58.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.224+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:00:58.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.224+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:00:58.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.225+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f401ac40 con 0x7f6400146050 2021-03-09T10:00:58.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:58.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:00:58.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:58.644+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:00:59.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:59.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:00:59.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:00:59.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:00.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:00.182+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:00.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:00.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:00.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:00.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:01.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:01.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:01.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:01.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:02.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:02.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:02.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:02.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:03.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:03.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:03.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11967 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4005180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:03.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11967 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4005180).stop 2021-03-09T10:01:03.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:03.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:03.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:03.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:03.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:01:03.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:03.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:03.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:03.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:03.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:03.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:03.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:03.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001910 con 0x7f63d8003400 2021-03-09T10:01:03.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:03.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce50 con 0x7f63d8008840 2021-03-09T10:01:03.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:03.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002abb0 con 0x7f63e0018390 2021-03-09T10:01:03.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.222+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:03.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.223+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:03.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.223+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:03.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.223+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:03.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.223+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:03.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.223+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:03.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.223+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:01:03.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.223+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:03.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.223+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:03.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:03.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:03.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:03.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:03.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:03.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:03.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:03.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:03.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:01:03.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:01:03.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:03.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:03.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007bf80 con 0x7f63e0018390 2021-03-09T10:01:03.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:03.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:03.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:01:03.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:01:03.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.224+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:03.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.225+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11982 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a7b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:03.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.225+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:01:03.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.225+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:03.241 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:03.227+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 32983 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:03.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.230+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc029290 con 0x7f63e0018390 2021-03-09T10:01:03.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.230+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:03.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.230+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:03.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.230+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:03.242 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:03.243 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:03.243 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:03.243 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:03.243 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:03.244 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:03.244 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:03.244 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:03.244 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:03.245 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.230+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0293f0 con 0x7f63e0018390 2021-03-09T10:01:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.230+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:03.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.230+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:01:03.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.230+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63e0018390 2021-03-09T10:01:03.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:03.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:03.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:04.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:04.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:04.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:04.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:05.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:05.183+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:05.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:05.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:05.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:05.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:06.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:06.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:06.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:06.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:07.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:07.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:07.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:07.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:08.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.227+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:08.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.227+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:08.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.227+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11982 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a7b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:08.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.227+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=11982 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a7b0).stop 2021-03-09T10:01:08.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:08.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:08.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:08.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=11982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:08.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:01:08.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:08.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:08.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:08.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:08.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:08.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:08.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:08.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002dc40 con 0x7f63d8003400 2021-03-09T10:01:08.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:08.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T10:01:08.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:08.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:01:08.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:08.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:08.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:08.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:08.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:08.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:08.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:01:08.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:08.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:08.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:08.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:08.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.228+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:08.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:08.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:08.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:08.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:08.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:08.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:01:08.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:01:08.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:08.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:08.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028490 con 0x7f6400146050 2021-03-09T10:01:08.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:08.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:08.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007bf80 con 0x7f6400146050 2021-03-09T10:01:08.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:01:08.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.229+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:08.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.230+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:01:08.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.230+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11995 cs=0 l=1 rev1=1 rx=0x7f63fc003a20 tx=0x7f63fc05df70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:08.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.230+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:08.245 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:08.233+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33021 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:08.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06fe80 con 0x7f6400146050 2021-03-09T10:01:08.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.233+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:08.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.233+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:08.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.233+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:08.247 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:08.247 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:08.247 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:08.248 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:08.248 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:08.248 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:08.248 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:08.249 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:08.249 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:08.249 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:08.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f6400146050 2021-03-09T10:01:08.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.233+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:08.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.233+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:01:08.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0473e0 con 0x7f6400146050 2021-03-09T10:01:08.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:08.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:08.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:08.645+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:09.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:09.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:09.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:09.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:10.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:10.184+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:10.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:10.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:10.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:10.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:11.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:11.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:11.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:11.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:12.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:12.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:12.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:12.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:13.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:13.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:13.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11995 cs=0 l=1 rev1=1 rx=0x7f63fc003a20 tx=0x7f63fc05df70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:13.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=11995 cs=0 l=1 rev1=1 rx=0x7f63fc003a20 tx=0x7f63fc05df70).stop 2021-03-09T10:01:13.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:13.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:13.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:13.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=11995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:13.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:01:13.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:13.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:13.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:13.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:13.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c3a0 con 0x7f63d8003400 2021-03-09T10:01:13.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:13.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000f070 con 0x7f63d8008840 2021-03-09T10:01:13.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:13.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e0018390 2021-03-09T10:01:13.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:13.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:13.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:13.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:13.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:13.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:13.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:01:13.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:13.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:13.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:13.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:13.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:13.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.234+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:13.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:13.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:13.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:13.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:13.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:01:13.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:01:13.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:13.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:13.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065af0 con 0x7f63e0018390 2021-03-09T10:01:13.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:13.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:13.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024c30 con 0x7f63e0018390 2021-03-09T10:01:13.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:01:13.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.235+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:13.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.236+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:01:13.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.236+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:13.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.236+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12009 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f00886c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:13.249 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:13.238+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33061 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:13.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.238+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:01:13.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.238+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:13.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.238+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:13.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.238+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:13.251 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:13.251 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:13.252 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:13.252 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:13.252 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:13.253 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:13.253 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:13.253 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:13.253 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:13.254 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:13.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.241+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:01:13.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.241+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:13.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.241+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:01:13.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.242+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:01:13.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:13.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:13.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:14.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:14.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:14.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:14.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:15.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:15.185+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:15.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:15.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:15.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:15.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:16.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:16.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:16.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:16.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:17.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:17.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:17.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:17.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:18.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.238+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:18.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.238+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:18.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.238+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12009 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f00886c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:18.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.238+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12009 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f00886c0).stop 2021-03-09T10:01:18.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:18.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:18.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:18.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:18.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:01:18.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:18.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:18.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:18.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:18.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:18.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:18.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:18.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b940 con 0x7f63d8003400 2021-03-09T10:01:18.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:18.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:01:18.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:18.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012fd0 con 0x7f6400146050 2021-03-09T10:01:18.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:18.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:18.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:18.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:18.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.239+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:18.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.240+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:18.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.240+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:01:18.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.240+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:18.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.240+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:18.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:18.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:18.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:18.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:18.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:18.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:18.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:18.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:18.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:01:18.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:01:18.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:18.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:18.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T10:01:18.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:18.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:18.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065af0 con 0x7f6400146050 2021-03-09T10:01:18.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:01:18.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.241+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:18.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.242+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12028 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007ffa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:18.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.242+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:01:18.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.242+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:18.262 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:18.243+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33099 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:18.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005ff00 con 0x7f6400146050 2021-03-09T10:01:18.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.243+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:18.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.243+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:18.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.243+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:18.264 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:18.264 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:18.264 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:18.264 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:18.265 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:18.265 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:18.265 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:18.265 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:18.266 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:18.266 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:18.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005ff00 con 0x7f6400146050 2021-03-09T10:01:18.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.245+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:18.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.245+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:01:18.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.246+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f6400146050 2021-03-09T10:01:18.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:18.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:18.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:18.646+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:19.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:19.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:19.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:19.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:20.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:20.186+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:20.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:20.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:20.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:20.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:21.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:21.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:21.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:21.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:22.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:22.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:22.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:22.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:23.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:23.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:23.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12028 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007ffa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12028 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007ffa0).stop 2021-03-09T10:01:23.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:23.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:23.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:23.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:23.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:01:23.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:23.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:23.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:23.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:23.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:23.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:23.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:23.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022560 con 0x7f63d8003400 2021-03-09T10:01:23.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:23.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8008840 2021-03-09T10:01:23.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:23.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63e0018390 2021-03-09T10:01:23.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.243+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:23.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:23.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:23.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:23.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:23.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:01:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:23.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:23.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:23.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:23.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:23.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:23.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:23.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:01:23.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:01:23.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:23.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:23.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:01:23.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:23.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:23.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T10:01:23.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:01:23.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.245+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:23.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.246+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:01:23.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.246+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12038 cs=0 l=1 rev1=1 rx=0x7f63f004dbe0 tx=0x7f63f0029f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:23.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.246+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:23.266 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:23.248+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33139 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:23.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.248+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005dcb0 con 0x7f63e0018390 2021-03-09T10:01:23.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.248+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:23.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.248+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:23.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.248+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:23.268 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:23.268 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:23.269 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:23.269 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:23.269 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:23.269 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:23.270 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:23.270 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:23.270 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:23.271 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:23.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.248+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63e0018390 2021-03-09T10:01:23.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.248+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:23.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.248+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:01:23.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63e0018390 2021-03-09T10:01:23.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:23.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:23.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:24.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:24.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:24.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:24.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:25.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:25.187+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:25.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:25.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:25.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:25.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:26.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:26.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:26.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:26.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:27.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:27.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:27.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:27.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:28.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:01:28.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:28.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12038 cs=0 l=1 rev1=1 rx=0x7f63f004dbe0 tx=0x7f63f0029f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:28.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12038 cs=0 l=1 rev1=1 rx=0x7f63f004dbe0 tx=0x7f63f0029f80).stop 2021-03-09T10:01:28.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:28.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:28.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:28.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:28.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:01:28.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:28.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:28.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:28.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:28.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:28.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:28.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:28.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012fd0 con 0x7f63d8003400 2021-03-09T10:01:28.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:28.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b940 con 0x7f63d8008840 2021-03-09T10:01:28.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:28.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:01:28.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.248+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:28.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:28.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:28.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.249+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:28.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.249+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:01:28.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.249+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:28.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.249+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:28.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.249+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:28.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.249+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:28.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.250+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:28.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.250+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:28.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.250+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:28.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:28.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:28.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:28.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:28.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:28.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:01:28.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:01:28.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:28.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:28.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T10:01:28.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:28.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:28.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028490 con 0x7f6400146050 2021-03-09T10:01:28.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:01:28.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.251+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:28.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.252+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12055 cs=0 l=1 rev1=1 rx=0x7f63e007c140 tx=0x7f63fc06b000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:28.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.252+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:01:28.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.252+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:28.264 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:28.253+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33177 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:28.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.253+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06fe80 con 0x7f6400146050 2021-03-09T10:01:28.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.253+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:28.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.253+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:28.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.253+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:28.266 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:28.267 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:28.267 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:28.267 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:28.267 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:28.267 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:28.268 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:28.268 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:28.268 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:28.268 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:28.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.258+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06fe80 con 0x7f6400146050 2021-03-09T10:01:28.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.258+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:28.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.258+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:01:28.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T10:01:28.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:28.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:28.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:28.648+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:29.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:29.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:29.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:29.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:30.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:30.188+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:30.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:30.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:30.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:31.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:31.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:31.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:31.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:32.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:32.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:32.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:32.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12055 cs=0 l=1 rev1=1 rx=0x7f63e007c140 tx=0x7f63fc06b000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:33.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12055 cs=0 l=1 rev1=1 rx=0x7f63e007c140 tx=0x7f63fc06b000).stop 2021-03-09T10:01:33.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:01:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:33.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.253+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:33.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:33.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:33.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:33.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:33.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:33.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d9a0 con 0x7f63d8003400 2021-03-09T10:01:33.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:33.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8008840 2021-03-09T10:01:33.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:33.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066990 con 0x7f63e0018390 2021-03-09T10:01:33.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:33.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:33.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:33.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:33.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:33.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:33.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.254+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:01:33.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.255+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:33.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.255+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:33.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.255+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:33.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.255+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:33.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.255+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:33.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.256+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:33.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.256+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:33.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.256+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:33.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.256+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:33.266 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:33.258+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33217 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:33.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.455+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:33.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.455+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T10:01:33.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.455+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:33.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.455+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T10:01:33.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:33.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:01:33.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:33.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:33.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:33.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:33.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:33.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:33.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:33.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:33.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:33.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:01:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:01:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cf50 con 0x7f63e0018390 2021-03-09T10:01:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:01:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:01:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:01:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12076 cs=0 l=1 rev1=1 rx=0x7f63f400edf0 tx=0x7f63f400ef40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.457+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022540 con 0x7f63e0018390 2021-03-09T10:01:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.458+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.458+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.458+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:33.468 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:33.468 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:33.468 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:33.468 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:33.469 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:33.469 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:33.469 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:33.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0018390 2021-03-09T10:01:33.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.458+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:33.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.458+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:01:33.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005b70 con 0x7f63e0018390 2021-03-09T10:01:33.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:33.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:33.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:34.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:34.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:34.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:34.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:35.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:35.189+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:35.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:35.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:35.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:35.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:36.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:36.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:36.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:37.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:37.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:37.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:37.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:38.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:38.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:38.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12076 cs=0 l=1 rev1=1 rx=0x7f63f400edf0 tx=0x7f63f400ef40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:38.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12076 cs=0 l=1 rev1=1 rx=0x7f63f400edf0 tx=0x7f63f400ef40).stop 2021-03-09T10:01:38.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:38.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:38.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:38.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:38.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:01:38.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:38.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:38.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:38.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:38.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:38.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:38.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:38.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8003400 2021-03-09T10:01:38.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:38.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:01:38.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:38.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012fd0 con 0x7f6400146050 2021-03-09T10:01:38.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:38.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:38.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.259+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:38.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.260+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:38.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.260+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:38.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.260+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:38.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.260+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:01:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.260+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.260+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.261+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.261+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.261+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:38.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:38.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:38.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:38.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:38.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:38.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:01:38.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:01:38.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:38.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:38.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b4f0 con 0x7f6400146050 2021-03-09T10:01:38.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:38.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:38.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T10:01:38.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:01:38.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.262+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:38.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.263+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12087 cs=0 l=1 rev1=1 rx=0x7f63e001b9e0 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:38.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.263+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:01:38.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.263+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:38.278 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:38.264+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33255 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:38.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052e80 con 0x7f6400146050 2021-03-09T10:01:38.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.265+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:38.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.265+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:38.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.265+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:38.280 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:38.281 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:38.281 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:38.282 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:38.282 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:38.282 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:38.284 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:38.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052e80 con 0x7f6400146050 2021-03-09T10:01:38.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.267+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.267+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:01:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T10:01:38.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:38.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:38.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:38.649+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:39.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:39.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:39.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:39.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:40.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:40.190+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:40.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:40.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:40.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:40.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:41.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:41.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:41.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:41.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:42.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:42.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:42.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:42.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.264+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.264+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:43.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.264+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12087 cs=0 l=1 rev1=1 rx=0x7f63e001b9e0 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.264+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12087 cs=0 l=1 rev1=1 rx=0x7f63e001b9e0 tx=0x7f63fc009a00).stop 2021-03-09T10:01:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:43.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12087 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:01:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:43.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:43.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:01:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018b30 con 0x7f63d8008840 2021-03-09T10:01:43.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:43.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63e0018390 2021-03-09T10:01:43.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:43.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:43.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:43.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:43.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:43.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:43.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:01:43.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:43.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:43.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:43.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:43.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:43.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:43.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:01:43.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:01:43.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:43.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:43.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007bf80 con 0x7f63e0018390 2021-03-09T10:01:43.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:43.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:43.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:01:43.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:01:43.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.266+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:43.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.267+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12093 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:43.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.267+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:01:43.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.267+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:43.282 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:43.269+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33295 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:43.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:01:43.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.270+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:43.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.270+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:43.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.270+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:43.284 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:43.284 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:43.284 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:43.286 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:43.286 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:43.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63e0018390 2021-03-09T10:01:43.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.274+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:43.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.274+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:01:43.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:01:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:43.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:43.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:44.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:44.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:44.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:45.191+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:45.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:45.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:45.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:45.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:46.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:46.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:46.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:47.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:47.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:47.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:47.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:48.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:01:48.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:48.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12093 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:48.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12093 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0006610).stop 2021-03-09T10:01:48.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:48.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:48.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:48.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:48.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:01:48.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:48.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:48.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:48.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:48.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:48.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:48.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:48.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012fd0 con 0x7f63d8003400 2021-03-09T10:01:48.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:48.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:01:48.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:48.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c3d0 con 0x7f6400146050 2021-03-09T10:01:48.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:48.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:48.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:48.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:48.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.270+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:48.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.271+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:48.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.271+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:01:48.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.271+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:48.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.271+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:48.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.271+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:48.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.271+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:48.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.271+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:48.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.272+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:48.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:48.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:48.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:48.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:48.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:01:48.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:01:48.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:48.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:48.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028490 con 0x7f6400146050 2021-03-09T10:01:48.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:48.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:48.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b4f0 con 0x7f6400146050 2021-03-09T10:01:48.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:01:48.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:48.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12113 cs=0 l=1 rev1=1 rx=0x7f63e0019e20 tx=0x7f63fc03de50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:48.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:01:48.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.273+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:48.287 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:48.275+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33333 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:48.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.276+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013eb0 con 0x7f6400146050 2021-03-09T10:01:48.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.276+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:48.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.276+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:48.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.276+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:48.289 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:48.289 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:48.289 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:48.289 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:48.290 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:48.290 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:48.290 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:48.290 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:48.291 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.278+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013eb0 con 0x7f6400146050 2021-03-09T10:01:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.278+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.278+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:01:48.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.281+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T10:01:48.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:48.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:48.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:48.650+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:49.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:49.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:49.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:49.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:50.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:50.192+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:50.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:50.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:50.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:50.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:51.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:51.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:51.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:51.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:52.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:52.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:52.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:52.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:53.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:53.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:53.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12113 cs=0 l=1 rev1=1 rx=0x7f63e0019e20 tx=0x7f63fc03de50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:53.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12113 cs=0 l=1 rev1=1 rx=0x7f63e0019e20 tx=0x7f63fc03de50).stop 2021-03-09T10:01:53.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:53.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:53.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:53.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:53.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:01:53.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:53.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:53.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0029270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:53.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:53.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:53.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:53.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:53.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63d8003400 2021-03-09T10:01:53.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:53.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fa00 con 0x7f63d8008840 2021-03-09T10:01:53.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:53.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017290 con 0x7f63e0018390 2021-03-09T10:01:53.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:53.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:53.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:53.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:53.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:53.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.276+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0029270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:53.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.277+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:01:53.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.277+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:53.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.277+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:53.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.277+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:53.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.277+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:53.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.277+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:53.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:53.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:53.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:53.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:53.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:53.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:01:53.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:01:53.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:53.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:53.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:01:53.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:53.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:53.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028490 con 0x7f63e0018390 2021-03-09T10:01:53.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:01:53.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.278+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:53.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.279+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0029270 secure :-1 s=READY pgs=12121 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:53.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.279+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:01:53.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.279+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:53.294 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:53.281+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33382 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:53.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.281+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03dc00 con 0x7f63e0018390 2021-03-09T10:01:53.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.281+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:53.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.281+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:53.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.281+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:53.295 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:53.296 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:53.296 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:53.296 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:53.296 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:53.296 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:53.297 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:53.297 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:53.297 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:53.297 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:53.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065a30 con 0x7f63e0018390 2021-03-09T10:01:53.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.284+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:53.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.284+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:01:53.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.285+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63e0018390 2021-03-09T10:01:53.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:53.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:53.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:54.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:54.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:54.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:54.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:55.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:55.193+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:01:55.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:55.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:55.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:55.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:56.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:56.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:56.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:56.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:57.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:57.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:57.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:57.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:58.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0029270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:01:58.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0029270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:01:58.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0029270 secure :-1 s=READY pgs=12121 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:01:58.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0029270 secure :-1 s=READY pgs=12121 cs=0 l=1 rev1=1 rx=0x7f63fc0337b0 tx=0x7f63fc009a00).stop 2021-03-09T10:01:58.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:58.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:01:58.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0029270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:01:58.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0029270 unknown :-1 s=CLOSED pgs=12121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:01:58.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:01:58.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:58.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:01:58.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:58.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:01:58.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:01:58.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:01:58.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:58.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f750 con 0x7f63d8003400 2021-03-09T10:01:58.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:58.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:01:58.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:01:58.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f6400146050 2021-03-09T10:01:58.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.281+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:01:58.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.282+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:01:58.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.282+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:58.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.282+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:01:58.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.282+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:01:58.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.282+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:01:58.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.282+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:01:58.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.282+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:01:58.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.282+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:01:58.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.283+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:01:58.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.283+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:01:58.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.283+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:01:58.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:01:58.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:58.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:58.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:01:58.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:01:58.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:01:58.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:01:58.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:58.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:58.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T10:01:58.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:01:58.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:01:58.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007bf80 con 0x7f6400146050 2021-03-09T10:01:58.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:01:58.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.284+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:58.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.285+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:01:58.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.285+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:01:58.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.285+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12140 cs=0 l=1 rev1=1 rx=0x7f63e0018360 tx=0x7f63f0028f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:01:58.307 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:01:58.287+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33420 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:01:58.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.287+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T10:01:58.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.287+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:01:58.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.287+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:01:58.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.287+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:01:58.311 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:01:58.311 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:01:58.312 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:58.312 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:01:58.313 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:01:58.313 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:01:58.313 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:01:58.314 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:01:58.314 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:01:58.315 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:01:58.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T10:01:58.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.290+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:01:58.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.291+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:01:58.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.291+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T10:01:58.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:58.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:01:58.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:58.652+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:01:59.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:59.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:01:59.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:01:59.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:00.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:00.194+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:00.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:00.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:00.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:00.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:01.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:01.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:01.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:01.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:02.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:02.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:02.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:02.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:03.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.287+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:03.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.287+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:03.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.287+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12140 cs=0 l=1 rev1=1 rx=0x7f63e0018360 tx=0x7f63f0028f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:03.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.287+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12140 cs=0 l=1 rev1=1 rx=0x7f63e0018360 tx=0x7f63f0028f30).stop 2021-03-09T10:02:03.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.287+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:03.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:03.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:03.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:03.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:02:03.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:03.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:03.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:03.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:03.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:03.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:03.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:03.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:02:03.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:03.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bca0 con 0x7f63d8008840 2021-03-09T10:02:03.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:03.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015990 con 0x7f63e0018390 2021-03-09T10:02:03.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:03.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:03.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:03.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:03.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:02:03.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:03.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.288+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:03.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.289+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:03.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.289+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:03.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.289+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:03.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.289+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:03.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.289+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:03.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.289+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:03.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.289+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:03.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.289+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:03.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:03.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:03.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:02:03.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:02:03.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:03.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:03.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:02:03.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:03.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:03.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:02:03.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:02:03.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:03.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.290+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12150 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:03.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.291+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:02:03.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.291+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:03.305 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:03.293+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33458 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:03.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013eb0 con 0x7f63e0018390 2021-03-09T10:02:03.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.293+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:03.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.293+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:03.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.293+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:03.307 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:03.307 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:03.307 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:03.308 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:03.308 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:03.308 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:03.308 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:03.309 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:03.309 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:03.309 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:03.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.294+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02be60 con 0x7f63e0018390 2021-03-09T10:02:03.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.294+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:03.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.294+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:02:03.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63e0018390 2021-03-09T10:02:03.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:03.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:03.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:04.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:04.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:04.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:04.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:05.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:05.195+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:05.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:05.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:05.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:05.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:06.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:06.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:06.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:06.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:07.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:07.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:07.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:07.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:08.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:08.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12150 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:08.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12150 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc04a6d0).stop 2021-03-09T10:02:08.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:08.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:08.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:08.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:08.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:02:08.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:08.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:08.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:08.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:08.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:08.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:08.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:08.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012fd0 con 0x7f63d8003400 2021-03-09T10:02:08.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:08.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:02:08.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:08.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f6400146050 2021-03-09T10:02:08.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.293+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:08.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:08.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:08.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:08.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:08.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:08.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:02:08.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:08.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:08.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:08.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:08.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.294+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:08.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:08.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:08.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:08.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:08.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:08.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:02:08.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:02:08.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:08.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:08.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028490 con 0x7f6400146050 2021-03-09T10:02:08.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:08.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:08.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T10:02:08.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:02:08.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:08.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12161 cs=0 l=1 rev1=1 rx=0x7f63f0042ff0 tx=0x7f63f0000cb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:08.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:02:08.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.295+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:08.310 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:08.298+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33498 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:08.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.298+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:02:08.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.298+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:08.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.298+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:08.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.298+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:08.312 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:08.312 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:08.312 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:08.313 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:08.313 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:08.314 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:08.314 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:08.314 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:08.314 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:08.314 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:08.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:02:08.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.299+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:08.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.299+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:02:08.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f006d2c0 con 0x7f6400146050 2021-03-09T10:02:08.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:08.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:08.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:08.653+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:09.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:09.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:09.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:09.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:10.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:10.196+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:10.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:10.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:10.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:10.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:11.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:11.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:11.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:11.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:12.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:12.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:12.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:12.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:13.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:13.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:13.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12161 cs=0 l=1 rev1=1 rx=0x7f63f0042ff0 tx=0x7f63f0000cb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:13.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12161 cs=0 l=1 rev1=1 rx=0x7f63f0042ff0 tx=0x7f63f0000cb0).stop 2021-03-09T10:02:13.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:13.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:13.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:13.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:13.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:02:13.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:13.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:13.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:13.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:13.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0009530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:13.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:13.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:13.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014ec0 con 0x7f63d8003400 2021-03-09T10:02:13.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:13.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018b30 con 0x7f63d8008840 2021-03-09T10:02:13.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:13.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63e0018390 2021-03-09T10:02:13.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:13.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:13.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:13.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:13.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.299+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:13.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0009530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:13.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:02:13.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:13.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:13.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:13.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:13.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:13.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:13.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:13.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:13.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:13.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:13.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:02:13.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:02:13.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:13.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:13.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b680 con 0x7f63e0018390 2021-03-09T10:02:13.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:13.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:13.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028490 con 0x7f63e0018390 2021-03-09T10:02:13.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:02:13.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.300+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:13.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.301+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:02:13.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.301+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:13.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0009530 secure :-1 s=READY pgs=12175 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0041ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:13.314 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:13.304+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33538 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:13.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036b40 con 0x7f63e0018390 2021-03-09T10:02:13.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.304+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:13.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.304+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:13.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.304+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:13.316 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:13.316 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:13.316 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:13.317 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:13.317 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:13.317 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:13.317 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:13.317 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:13.318 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:13.318 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:13.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0036b40 con 0x7f63e0018390 2021-03-09T10:02:13.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.304+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:13.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.304+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:02:13.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.306+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00036b0 con 0x7f63e0018390 2021-03-09T10:02:13.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:13.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:13.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:14.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:14.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:14.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:14.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:15.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:15.197+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:15.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:15.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:15.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:15.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:16.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:16.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:16.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:16.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:17.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:17.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:17.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:17.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:18.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0009530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:18.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0009530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:18.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0009530 secure :-1 s=READY pgs=12175 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0041ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:18.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0009530 secure :-1 s=READY pgs=12175 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0041ec0).stop 2021-03-09T10:02:18.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:18.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:18.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0009530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:18.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0009530 unknown :-1 s=CLOSED pgs=12175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:18.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:02:18.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:18.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:18.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:18.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:18.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:18.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:18.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:18.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:02:18.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:18.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bca0 con 0x7f63d8008840 2021-03-09T10:02:18.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:18.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c780 con 0x7f6400146050 2021-03-09T10:02:18.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:18.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:18.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:18.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:18.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.305+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:18.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.306+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:18.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.306+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:02:18.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.306+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:18.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.306+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:18.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.306+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:18.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.306+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:18.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.306+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:18.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:18.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:18.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:18.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:18.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:18.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:02:18.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:02:18.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:18.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:18.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T10:02:18.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:18.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:18.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cf50 con 0x7f6400146050 2021-03-09T10:02:18.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:02:18.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.307+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:18.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.308+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:02:18.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.308+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:18.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.308+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12189 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4017f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:18.321 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:18.310+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33576 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:18.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40462c0 con 0x7f6400146050 2021-03-09T10:02:18.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.311+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:18.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.311+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:18.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.311+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:18.323 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:18.323 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:18.324 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:18.324 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:18.324 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:18.324 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:18.324 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:18.325 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:18.325 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:18.325 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:18.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f6400146050 2021-03-09T10:02:18.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.314+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:18.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.314+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:02:18.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.315+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40462c0 con 0x7f6400146050 2021-03-09T10:02:18.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:18.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:18.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:18.654+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:19.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:19.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:19.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:19.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:20.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:20.198+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:20.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:20.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:20.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:20.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:21.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:21.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:21.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:21.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:22.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:22.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:22.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:22.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:23.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:23.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:23.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12189 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4017f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:23.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12189 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f4017f40).stop 2021-03-09T10:02:23.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:23.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:23.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:23.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:23.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:02:23.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:23.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:23.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:23.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:23.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:23.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:23.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:23.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:02:23.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:23.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018b30 con 0x7f63d8008840 2021-03-09T10:02:23.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:23.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0018390 2021-03-09T10:02:23.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:23.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:23.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:23.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:23.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:23.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.311+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:23.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:02:23.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:23.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:23.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:23.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:23.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:23.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:23.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:23.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:23.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:23.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:23.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:02:23.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:02:23.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:23.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:23.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028490 con 0x7f63e0018390 2021-03-09T10:02:23.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:23.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:23.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b680 con 0x7f63e0018390 2021-03-09T10:02:23.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:02:23.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.312+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:23.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.314+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12205 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:23.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.314+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:02:23.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.314+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:23.327 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:23.316+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33614 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:23.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f4c0 con 0x7f63e0018390 2021-03-09T10:02:23.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.316+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:23.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.316+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:23.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.316+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:23.329 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:23.329 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:23.330 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:23.330 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:23.330 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:23.330 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:23.330 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:23.331 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:23.331 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:23.331 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:23.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63e0018390 2021-03-09T10:02:23.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.318+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:23.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.320+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:02:23.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.320+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:02:23.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:23.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:23.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:24.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:24.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:24.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:24.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:25.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:25.199+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:25.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:25.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:25.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:25.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:26.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:26.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:26.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:26.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:27.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:27.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:27.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:27.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:28.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:28.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:28.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12205 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:28.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12205 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004dd90).stop 2021-03-09T10:02:28.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:28.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:28.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:28.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:28.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:02:28.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:28.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:28.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:28.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:28.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:28.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:28.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:28.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:02:28.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:28.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8008840 2021-03-09T10:02:28.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:28.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f6400146050 2021-03-09T10:02:28.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:28.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:28.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:28.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:28.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:28.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:28.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:02:28.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:28.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.317+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:28.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.318+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:28.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.318+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:28.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.318+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:28.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:28.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:28.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:28.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:28.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:28.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:02:28.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:02:28.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:28.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:28.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d60 con 0x7f6400146050 2021-03-09T10:02:28.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:28.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:28.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f6400146050 2021-03-09T10:02:28.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:02:28.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.319+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:28.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.320+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:02:28.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.320+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:28.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.320+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12221 cs=0 l=1 rev1=1 rx=0x7f63fc012720 tx=0x7f63fc06ddd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:28.334 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:28.321+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33652 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:28.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.322+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f6400146050 2021-03-09T10:02:28.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.322+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:28.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.322+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:28.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.322+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:28.335 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:28.336 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:28.336 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:28.336 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:28.336 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:28.337 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:28.337 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:28.337 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:28.337 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:28.337 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:28.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f6400146050 2021-03-09T10:02:28.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.328+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:28.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.328+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:02:28.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013da0 con 0x7f6400146050 2021-03-09T10:02:28.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:28.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:28.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:28.655+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:29.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:29.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:29.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:29.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:30.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:30.200+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:30.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:30.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:30.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:30.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:31.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:31.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:31.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:32.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:32.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:32.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:32.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:33.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:33.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:33.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12221 cs=0 l=1 rev1=1 rx=0x7f63fc012720 tx=0x7f63fc06ddd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:33.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12221 cs=0 l=1 rev1=1 rx=0x7f63fc012720 tx=0x7f63fc06ddd0).stop 2021-03-09T10:02:33.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:33.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:33.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:33.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:33.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:02:33.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:33.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:33.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:33.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:33.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:33.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:33.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:33.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f1e0 con 0x7f63d8003400 2021-03-09T10:02:33.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:33.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012fd0 con 0x7f63d8008840 2021-03-09T10:02:33.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:33.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:02:33.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:33.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:33.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.322+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:33.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.323+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:33.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.323+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:33.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.323+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:33.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.323+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:02:33.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.323+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:33.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.323+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:33.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:33.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:33.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:33.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:33.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:33.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:33.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:33.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:33.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:02:33.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:02:33.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:33.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:33.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002abc0 con 0x7f63e0018390 2021-03-09T10:02:33.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:33.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:33.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.324+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028490 con 0x7f63e0018390 2021-03-09T10:02:33.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.325+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:02:33.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.325+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:33.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.325+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:02:33.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.325+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:33.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.325+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065600 secure :-1 s=READY pgs=12234 cs=0 l=1 rev1=1 rx=0x7f63f006d460 tx=0x7f63f006d330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:33.338 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:33.327+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33694 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:33.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63e0018390 2021-03-09T10:02:33.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.328+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:33.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.328+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:33.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.328+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:33.340 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:33.340 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:33.340 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:33.340 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:33.341 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:33.341 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:33.341 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:33.341 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:33.341 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:33.342 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:33.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63e0018390 2021-03-09T10:02:33.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.331+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:33.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.331+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:02:33.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.332+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:02:33.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:33.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:33.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:34.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:34.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:34.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:34.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:35.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:35.201+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:35.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:35.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:35.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:35.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:36.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:36.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:36.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:36.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:37.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:37.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:37.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:37.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:38.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0065600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:38.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0065600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:38.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065600 secure :-1 s=READY pgs=12234 cs=0 l=1 rev1=1 rx=0x7f63f006d460 tx=0x7f63f006d330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:38.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065600 secure :-1 s=READY pgs=12234 cs=0 l=1 rev1=1 rx=0x7f63f006d460 tx=0x7f63f006d330).stop 2021-03-09T10:02:38.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:38.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:38.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0065600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:38.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065600 unknown :-1 s=CLOSED pgs=12234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:38.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:02:38.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:38.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:38.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:38.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:38.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:38.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:38.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:38.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005edf0 con 0x7f63d8003400 2021-03-09T10:02:38.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:38.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8008840 2021-03-09T10:02:38.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:38.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T10:02:38.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:38.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:38.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:38.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:38.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.328+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:38.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:38.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.329+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:02:38.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.329+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:38.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.329+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:38.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.329+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:38.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.329+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:38.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.329+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:38.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:38.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:38.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:38.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:38.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:38.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:02:38.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:02:38.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:38.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:38.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025490 con 0x7f6400146050 2021-03-09T10:02:38.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:38.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:38.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002abc0 con 0x7f6400146050 2021-03-09T10:02:38.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:02:38.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.331+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:38.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.332+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:02:38.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.332+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:38.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.332+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12252 cs=0 l=1 rev1=1 rx=0x7f63e004c240 tx=0x7f63f005f2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:38.343 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:38.333+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33732 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:38.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0042530 con 0x7f6400146050 2021-03-09T10:02:38.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.334+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:38.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.334+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:38.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.334+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:38.345 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:38.345 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:38.345 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:38.346 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:38.346 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:38.346 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:38.346 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:38.347 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:38.347 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:38.347 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:38.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0042530 con 0x7f6400146050 2021-03-09T10:02:38.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.338+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:38.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005eb10 con 0x7f6400146050 2021-03-09T10:02:38.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.338+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:02:38.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:38.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:38.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:38.657+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:39.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:39.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:39.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:39.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:40.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:40.202+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:40.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:40.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:40.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:40.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:41.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:41.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:41.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:41.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:42.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:42.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:42.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:42.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:43.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:43.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:43.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12252 cs=0 l=1 rev1=1 rx=0x7f63e004c240 tx=0x7f63f005f2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:43.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12252 cs=0 l=1 rev1=1 rx=0x7f63e004c240 tx=0x7f63f005f2e0).stop 2021-03-09T10:02:43.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:43.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:43.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:43.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:43.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:02:43.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:43.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:43.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:43.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:43.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:43.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:43.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:43.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018ff0 con 0x7f63d8003400 2021-03-09T10:02:43.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:43.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T10:02:43.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:43.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63e0018390 2021-03-09T10:02:43.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:43.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:43.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:43.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:43.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.334+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.335+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.335+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:02:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.335+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:43.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.335+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:43.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.335+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:43.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.335+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:43.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.335+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:43.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:43.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:43.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:43.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:43.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:43.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:02:43.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:02:43.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:43.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:43.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T10:02:43.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:43.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:43.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025490 con 0x7f63e0018390 2021-03-09T10:02:43.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:02:43.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.336+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:43.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.337+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12265 cs=0 l=1 rev1=1 rx=0x7f63f007e7e0 tx=0x7f63f0030300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:43.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.338+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:02:43.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.338+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:43.349 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:43.338+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33772 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:43.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.339+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e620 con 0x7f63e0018390 2021-03-09T10:02:43.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.339+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:43.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.339+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:43.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.339+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:43.351 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:43.351 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:43.352 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:43.352 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:43.352 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:43.352 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:43.352 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:43.353 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:43.353 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:43.353 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:43.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e620 con 0x7f63e0018390 2021-03-09T10:02:43.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.341+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:43.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.341+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:02:43.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63e0018390 2021-03-09T10:02:43.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:43.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:43.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:44.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:44.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:44.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:44.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:45.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:45.203+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:45.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:45.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:45.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:45.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:46.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:46.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:46.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:47.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:47.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:47.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:47.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:48.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:02:48.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:48.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12265 cs=0 l=1 rev1=1 rx=0x7f63f007e7e0 tx=0x7f63f0030300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12265 cs=0 l=1 rev1=1 rx=0x7f63f007e7e0 tx=0x7f63f0030300).stop 2021-03-09T10:02:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:48.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:02:48.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:48.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:48.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:48.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:48.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:48.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:48.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:48.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:02:48.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:48.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009910 con 0x7f63d8008840 2021-03-09T10:02:48.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:48.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f6400146050 2021-03-09T10:02:48.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.339+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:48.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.340+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:48.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.340+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:48.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:48.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.340+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:48.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.340+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:48.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.340+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:02:48.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.340+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:48.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.340+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:48.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:48.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:48.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:48.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:48.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:48.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:02:48.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:02:48.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:48.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:48.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010cf0 con 0x7f6400146050 2021-03-09T10:02:48.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:48.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:48.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d60 con 0x7f6400146050 2021-03-09T10:02:48.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:02:48.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.343+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:48.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.344+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12279 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc065f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:48.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.344+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:02:48.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.344+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:48.360 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:48.344+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33810 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:48.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013da0 con 0x7f6400146050 2021-03-09T10:02:48.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.345+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:48.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.345+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:48.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.345+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:48.362 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:48.362 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:48.362 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:48.363 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:48.363 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:48.363 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:48.364 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:48.364 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:48.364 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:48.365 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:48.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013da0 con 0x7f6400146050 2021-03-09T10:02:48.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.346+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:48.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.346+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:02:48.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013da0 con 0x7f6400146050 2021-03-09T10:02:48.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:48.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:48.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:48.658+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:49.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:49.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:49.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:49.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:50.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:50.204+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:50.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:50.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:50.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:50.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:51.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:51.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:51.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:51.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:52.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:52.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:52.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:52.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:53.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:53.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:53.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12279 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc065f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:53.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12279 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc065f40).stop 2021-03-09T10:02:53.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:53.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:53.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:53.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:53.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:02:53.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:53.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:53.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:53.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:53.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:53.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:53.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:53.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079330 con 0x7f63d8003400 2021-03-09T10:02:53.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:53.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ca70 con 0x7f63d8008840 2021-03-09T10:02:53.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:53.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63e0018390 2021-03-09T10:02:53.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:53.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:53.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:53.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:53.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:53.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:53.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:02:53.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.345+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:53.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:53.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:53.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:53.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:53.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:53.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:53.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:53.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:53.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:53.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:02:53.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:02:53.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:53.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:53.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:02:53.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:53.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:53.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.346+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010cf0 con 0x7f63e0018390 2021-03-09T10:02:53.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.347+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:02:53.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.347+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:53.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.347+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:02:53.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.347+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:53.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.347+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12287 cs=0 l=1 rev1=1 rx=0x7f63e002b790 tx=0x7f63fc04bd20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:53.363 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:53.350+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33850 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:53.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012c10 con 0x7f63e0018390 2021-03-09T10:02:53.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.352+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:53.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.352+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:53.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.352+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:53.365 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:53.365 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:53.365 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:53.365 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:53.366 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:53.366 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:53.366 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:53.366 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:53.367 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:53.367 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:53.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.353+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc012c10 con 0x7f63e0018390 2021-03-09T10:02:53.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.353+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:53.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.354+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:02:53.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.354+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc04b9c0 con 0x7f63e0018390 2021-03-09T10:02:53.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:53.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:53.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:54.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:54.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:54.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:54.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:55.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:55.205+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:02:55.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:55.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:55.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:55.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:56.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:56.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:56.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:56.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:57.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:57.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:57.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:57.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:58.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:02:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:02:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12287 cs=0 l=1 rev1=1 rx=0x7f63e002b790 tx=0x7f63fc04bd20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:02:58.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12287 cs=0 l=1 rev1=1 rx=0x7f63e002b790 tx=0x7f63fc04bd20).stop 2021-03-09T10:02:58.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:58.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:02:58.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:02:58.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:02:58.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:02:58.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:58.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:02:58.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:58.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:02:58.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:02:58.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:02:58.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:58.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00256e0 con 0x7f63d8003400 2021-03-09T10:02:58.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:58.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00150c0 con 0x7f63d8008840 2021-03-09T10:02:58.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:02:58.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:02:58.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:02:58.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:02:58.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:58.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:02:58.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:02:58.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:02:58.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:02:58.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:02:58.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:02:58.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:02:58.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:02:58.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.351+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:02:58.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:02:58.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:58.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:58.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:02:58.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:02:58.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:02:58.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:02:58.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:58.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:58.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016bf0 con 0x7f6400146050 2021-03-09T10:02:58.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:02:58.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:02:58.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T10:02:58.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:02:58.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.352+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:58.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.353+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:02:58.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.353+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:02:58.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.353+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12300 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f005f350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:02:58.367 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:02:58.355+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33888 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:02:58.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f6400146050 2021-03-09T10:02:58.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.356+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:02:58.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.356+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:02:58.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.356+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:02:58.369 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:02:58.369 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:02:58.369 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:58.370 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:02:58.370 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:02:58.370 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:02:58.370 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:02:58.371 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:02:58.371 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:02:58.371 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:02:58.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.359+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f6400146050 2021-03-09T10:02:58.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.359+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:02:58.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.359+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:02:58.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.361+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T10:02:58.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:58.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:02:58.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:58.659+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:02:59.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:59.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:02:59.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:02:59.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:00.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:00.206+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:00.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:00.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:00.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:00.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:01.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:01.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:01.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:01.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:02.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:02.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:02.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:02.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:03.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:03.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:03.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12300 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f005f350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:03.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12300 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f005f350).stop 2021-03-09T10:03:03.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:03.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:03.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:03:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:03.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:03.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:03:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011fb0 con 0x7f63d8008840 2021-03-09T10:03:03.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067c10 con 0x7f63e0018390 2021-03-09T10:03:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:03.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:03.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.357+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.357+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:03:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.357+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:03.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.357+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.357+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.357+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.357+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:03.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:03.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:03.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:03:03.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:03:03.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:03.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:03.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025490 con 0x7f63e0018390 2021-03-09T10:03:03.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:03.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:03.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:03:03.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:03:03.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.359+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:03.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.360+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:03:03.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.360+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:03.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.360+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12319 cs=0 l=1 rev1=1 rx=0x7f63e0065450 tx=0x7f63fc075fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:03.373 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:03.361+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33928 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:03.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06fc60 con 0x7f63e0018390 2021-03-09T10:03:03.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.362+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:03.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.362+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:03.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.362+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:03.375 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:03.376 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:03.376 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:03.376 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:03.376 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:03.376 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:03.377 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:03.377 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:03.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06fc60 con 0x7f63e0018390 2021-03-09T10:03:03.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.365+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:03.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.365+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:03:03.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.366+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63e0018390 2021-03-09T10:03:03.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:03.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:03.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:04.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:04.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:04.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:04.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:05.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:05.207+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:05.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:05.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:05.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:05.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:06.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:06.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:06.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:06.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:07.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:07.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:08.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:08.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:08.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12319 cs=0 l=1 rev1=1 rx=0x7f63e0065450 tx=0x7f63fc075fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:08.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12319 cs=0 l=1 rev1=1 rx=0x7f63e0065450 tx=0x7f63fc075fe0).stop 2021-03-09T10:03:08.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:08.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:08.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:08.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:03:08.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:08.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:08.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:08.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:08.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:08.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:08.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:08.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a950 con 0x7f63d8003400 2021-03-09T10:03:08.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:08.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ca70 con 0x7f63d8008840 2021-03-09T10:03:08.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:08.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066160 con 0x7f6400146050 2021-03-09T10:03:08.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:08.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:08.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:08.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:08.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:08.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:08.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:03:08.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:08.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.362+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:08.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:08.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:08.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:08.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:08.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:08.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:08.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:08.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:08.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:03:08.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:03:08.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:08.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:08.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010cf0 con 0x7f6400146050 2021-03-09T10:03:08.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:08.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:08.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005de90 con 0x7f6400146050 2021-03-09T10:03:08.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:03:08.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.363+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:08.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.364+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:03:08.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.364+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12327 cs=0 l=1 rev1=1 rx=0x7f63f4005270 tx=0x7f63f4003b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:08.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.364+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:08.392 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:08.367+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 33964 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:08.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005ce0 con 0x7f6400146050 2021-03-09T10:03:08.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.368+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:08.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.368+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:08.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.368+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:08.395 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:08.396 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:08.396 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:08.397 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:08.397 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:08.397 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:08.398 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:08.398 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:08.399 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:08.399 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:08.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022870 con 0x7f6400146050 2021-03-09T10:03:08.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.368+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.368+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:03:08.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.369+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40229e0 con 0x7f6400146050 2021-03-09T10:03:08.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:08.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:08.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:08.660+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:09.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:09.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:09.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:09.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:10.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:10.208+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:10.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:10.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:10.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:11.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:11.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:11.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:11.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:12.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:12.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:12.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:12.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:13.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:13.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:13.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12327 cs=0 l=1 rev1=1 rx=0x7f63f4005270 tx=0x7f63f4003b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:13.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12327 cs=0 l=1 rev1=1 rx=0x7f63f4005270 tx=0x7f63f4003b40).stop 2021-03-09T10:03:13.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:13.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:13.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:13.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:13.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:03:13.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:13.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:13.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:13.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:13.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00299c0 con 0x7f63d8003400 2021-03-09T10:03:13.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:13.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:03:13.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:13.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac10 con 0x7f63e0018390 2021-03-09T10:03:13.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:13.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:13.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:13.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:13.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:13.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:13.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:03:13.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.368+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:13.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.369+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:13.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.369+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:13.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.369+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:13.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.369+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:13.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.369+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:13.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.369+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:13.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.369+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:13.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:13.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:13.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:03:13.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:03:13.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:13.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:13.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028bb0 con 0x7f63e0018390 2021-03-09T10:03:13.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:13.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:13.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016bf0 con 0x7f63e0018390 2021-03-09T10:03:13.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:03:13.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:13.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:03:13.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12341 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0029fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:13.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.370+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:13.387 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:13.373+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34006 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:13.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.373+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:03:13.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.373+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:13.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.373+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:13.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.373+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:13.389 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:13.389 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:13.389 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:13.390 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:13.390 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:13.390 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:13.390 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:13.391 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:13.391 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:13.391 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:13.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:03:13.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.374+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:13.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.374+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:03:13.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029bd0 con 0x7f63e0018390 2021-03-09T10:03:13.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:13.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:13.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:14.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:14.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:14.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:14.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:15.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:15.209+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:15.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:15.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:15.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:15.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:16.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:16.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:16.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:16.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:17.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:17.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:17.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:17.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:18.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:18.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:18.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12341 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0029fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:18.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12341 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0029fb0).stop 2021-03-09T10:03:18.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:18.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:18.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:18.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:18.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:03:18.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:18.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:18.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:18.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:18.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:18.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:18.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:18.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022790 con 0x7f63d8003400 2021-03-09T10:03:18.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:18.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064ef0 con 0x7f63d8008840 2021-03-09T10:03:18.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:18.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f6400146050 2021-03-09T10:03:18.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:18.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:18.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:18.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:18.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:18.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.374+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:18.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:03:18.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:18.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:18.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:18.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:18.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:18.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:18.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:18.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:18.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:18.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:18.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:03:18.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:03:18.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:18.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:18.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T10:03:18.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:18.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:18.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.375+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025490 con 0x7f6400146050 2021-03-09T10:03:18.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.376+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:03:18.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.376+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:18.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.376+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:03:18.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.376+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12354 cs=0 l=1 rev1=1 rx=0x7f63e00654b0 tx=0x7f63fc029dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:18.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.376+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:18.390 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:18.379+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34044 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:18.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f6400146050 2021-03-09T10:03:18.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.379+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:18.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.379+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:18.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.379+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:18.391 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:18.392 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:18.393 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:18.393 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:18.393 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:18.393 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:18.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f6400146050 2021-03-09T10:03:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.379+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.379+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:03:18.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc062780 con 0x7f6400146050 2021-03-09T10:03:18.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:18.662+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:19.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:19.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:19.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:19.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:20.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:20.210+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:20.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:20.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:20.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:20.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:21.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:21.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:21.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:22.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:22.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:22.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:22.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:23.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:23.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:23.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12354 cs=0 l=1 rev1=1 rx=0x7f63e00654b0 tx=0x7f63fc029dc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12354 cs=0 l=1 rev1=1 rx=0x7f63e00654b0 tx=0x7f63fc029dc0).stop 2021-03-09T10:03:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:23.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:03:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:23.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:23.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:23.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067c10 con 0x7f63d8003400 2021-03-09T10:03:23.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:23.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:03:23.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:23.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007bde0 con 0x7f63e0018390 2021-03-09T10:03:23.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:23.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:23.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:23.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:23.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:23.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:23.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:03:23.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:23.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:23.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:23.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:23.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.380+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:23.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:23.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:23.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:23.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:23.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:23.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:03:23.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:03:23.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:23.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:23.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:03:23.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:23.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:23.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010cf0 con 0x7f63e0018390 2021-03-09T10:03:23.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:03:23.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.381+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:23.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.383+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:03:23.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.383+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:23.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.383+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12376 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4005d50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:23.395 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:23.384+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34084 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.384+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40727a0 con 0x7f63e0018390 2021-03-09T10:03:23.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.384+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.384+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.384+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:23.397 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:23.398 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:23.398 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:23.398 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:23.398 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:23.399 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40349c0 con 0x7f63e0018390 2021-03-09T10:03:23.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.386+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:23.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.386+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:03:23.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40727a0 con 0x7f63e0018390 2021-03-09T10:03:23.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:23.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:23.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:24.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:24.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:24.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:25.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:25.211+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:25.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:25.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:25.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:25.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:26.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:26.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:26.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:26.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:27.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:27.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:27.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:27.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:28.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:28.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:28.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12376 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4005d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:28.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 secure :-1 s=READY pgs=12376 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4005d50).stop 2021-03-09T10:03:28.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:28.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:28.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:28.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e70 unknown :-1 s=CLOSED pgs=12376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:28.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:03:28.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:28.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:28.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:28.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:28.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:28.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:28.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:28.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e4d0 con 0x7f63d8003400 2021-03-09T10:03:28.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:28.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015830 con 0x7f63d8008840 2021-03-09T10:03:28.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:28.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018150 con 0x7f6400146050 2021-03-09T10:03:28.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:28.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:28.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.386+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:28.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.387+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:28.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.387+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:28.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.387+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:28.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.387+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:03:28.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.387+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:28.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.387+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:28.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.387+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:28.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.387+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:28.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.387+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:28.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:28.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:28.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:28.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:28.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:28.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:03:28.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:03:28.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:28.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:28.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T10:03:28.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:28.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:28.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028bb0 con 0x7f6400146050 2021-03-09T10:03:28.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:03:28.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.389+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:28.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.390+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:03:28.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.390+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:28.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.390+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12391 cs=0 l=1 rev1=1 rx=0x7f63f0028600 tx=0x7f63f00066f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:28.402 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:28.390+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34112 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:28.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.391+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007fdc0 con 0x7f6400146050 2021-03-09T10:03:28.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.391+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:28.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.391+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:28.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.391+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:28.404 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:28.404 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:28.404 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:28.405 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:28.405 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:28.405 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:28.405 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:28.405 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:28.406 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:28.406 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:28.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.394+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007fdc0 con 0x7f6400146050 2021-03-09T10:03:28.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.394+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:28.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.394+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:03:28.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.395+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f6400146050 2021-03-09T10:03:28.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:28.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:28.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:28.663+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.347+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 msgr2=0x7f63e005e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000836) for more than 900000000 us, fault. 2021-03-09T10:03:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.347+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e005e020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0006b30).stop 2021-03-09T10:03:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 msgr2=0x7f63e005e020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e005e020 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.347+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0020e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.347+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0020e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.347+0000 7f6405551700 10 monclient: get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:03:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.347+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:03:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.347+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0020e50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4005d50).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:29.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:29.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:29.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:30.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:30.212+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:30.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:30.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:30.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:30.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:31.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:31.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:31.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:31.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:32.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:32.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:32.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:32.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:33.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:03:33.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:33.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12391 cs=0 l=1 rev1=1 rx=0x7f63f0028600 tx=0x7f63f00066f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:33.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12391 cs=0 l=1 rev1=1 rx=0x7f63f0028600 tx=0x7f63f00066f0).stop 2021-03-09T10:03:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:33.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:33.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:33.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:03:33.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:33.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:33.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:33.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:33.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:33.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013f40 con 0x7f63d8003400 2021-03-09T10:03:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:03:33.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:33.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005edf0 con 0x7f63d8005e00 2021-03-09T10:03:33.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.391+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:33.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:33.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:33.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:33.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:33.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:33.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:03:33.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:33.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:33.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:33.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:33.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.392+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:33.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:33.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:33.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:33.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:33.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:33.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:03:33.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:03:33.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:33.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:33.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c5d0 con 0x7f63d8005e00 2021-03-09T10:03:33.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:33.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:33.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T10:03:33.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:03:33.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.393+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:33.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.394+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005ded0 secure :-1 s=READY pgs=12398 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:33.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.394+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:03:33.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.394+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:33.418 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:33.396+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34162 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:33.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc029e30 con 0x7f63d8005e00 2021-03-09T10:03:33.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.396+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:33.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.396+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:33.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.396+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:33.421 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:33.422 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:33.422 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:33.423 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:33.423 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:33.423 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:33.424 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:33.424 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:33.425 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:33.425 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:33.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071120 con 0x7f63d8005e00 2021-03-09T10:03:33.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.397+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:33.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.397+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:03:33.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.398+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065750 con 0x7f63d8005e00 2021-03-09T10:03:33.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:33.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:33.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:34.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:34.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:34.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:34.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:35.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:35.213+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:35.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:35.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:35.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:35.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:36.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:36.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:36.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:36.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:37.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:37.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:37.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:37.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:38.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e005ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:38.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e005ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:38.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005ded0 secure :-1 s=READY pgs=12398 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:38.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005ded0 secure :-1 s=READY pgs=12398 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc003a50).stop 2021-03-09T10:03:38.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:38.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:38.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e005ded0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:38.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005ded0 unknown :-1 s=CLOSED pgs=12398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:38.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:03:38.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:38.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:38.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:38.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:38.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:38.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:38.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:38.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T10:03:38.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:38.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022790 con 0x7f63d8008840 2021-03-09T10:03:38.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:38.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002cb0 con 0x7f6400146050 2021-03-09T10:03:38.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:38.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:38.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:38.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:38.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:38.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:38.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:03:38.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:38.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.397+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:38.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:38.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:38.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:38.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:38.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:38.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:38.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:38.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:38.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:03:38.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:03:38.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:38.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:38.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0069300 con 0x7f6400146050 2021-03-09T10:03:38.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:38.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:38.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T10:03:38.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:03:38.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.398+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:38.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.399+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12412 cs=0 l=1 rev1=1 rx=0x7f63f0063ff0 tx=0x7f63f0013bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:38.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.399+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:03:38.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.399+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:38.412 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:38.402+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34200 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f6400146050 2021-03-09T10:03:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.403+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.403+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:38.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.403+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:38.414 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:38.415 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:38.416 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0038e80 con 0x7f6400146050 2021-03-09T10:03:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.409+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:38.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.409+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:03:38.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T10:03:38.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:38.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:38.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:38.664+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:39.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:39.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:39.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:39.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:40.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:40.214+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:40.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:40.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:40.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:40.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:41.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:41.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:41.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:41.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:42.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:42.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:42.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:42.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:43.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.402+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:43.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.402+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:43.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.402+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12412 cs=0 l=1 rev1=1 rx=0x7f63f0063ff0 tx=0x7f63f0013bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:43.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.402+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12412 cs=0 l=1 rev1=1 rx=0x7f63f0063ff0 tx=0x7f63f0013bc0).stop 2021-03-09T10:03:43.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.402+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:43.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.402+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:43.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.402+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:43.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.402+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:43.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.402+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:03:43.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:43.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:43.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:43.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:43.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:43.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:43.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:43.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065800 con 0x7f63d8003400 2021-03-09T10:03:43.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:43.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8008840 2021-03-09T10:03:43.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:43.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:03:43.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:43.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:43.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:43.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:43.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.403+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:43.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.404+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:43.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.404+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:03:43.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.404+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:43.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.404+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:43.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.404+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:43.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.404+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:43.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.404+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:43.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:43.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:43.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:43.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:43.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:43.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:03:43.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:03:43.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:43.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:43.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T10:03:43.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:43.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:43.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0069300 con 0x7f63d8005e00 2021-03-09T10:03:43.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:03:43.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.405+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:43.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.406+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:03:43.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.406+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005e5d0 secure :-1 s=READY pgs=12434 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:43.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.406+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:43.420 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:43.408+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34240 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:43.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.408+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b4c0 con 0x7f63d8005e00 2021-03-09T10:03:43.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.408+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:43.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.408+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:43.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.408+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:43.421 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:43.422 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:43.422 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:43.422 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:43.422 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:43.423 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:43.423 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:43.423 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:43.423 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:43.423 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:43.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001c8e0 con 0x7f63d8005e00 2021-03-09T10:03:43.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.413+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:43.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.413+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:03:43.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.414+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:03:43.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:43.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:43.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:44.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:44.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:44.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:44.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:45.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:45.215+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:45.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:45.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:45.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:45.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:46.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:46.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:46.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:46.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:47.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:47.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:47.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:47.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:48.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.408+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e005e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:03:48.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.408+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e005e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:48.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.408+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005e5d0 secure :-1 s=READY pgs=12434 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:48.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005e5d0 secure :-1 s=READY pgs=12434 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f004db20).stop 2021-03-09T10:03:48.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:48.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:48.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e005e5d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:48.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005e5d0 unknown :-1 s=CLOSED pgs=12434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:48.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:03:48.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:48.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:48.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:48.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:48.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:48.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:48.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:48.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T10:03:48.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:48.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010500 con 0x7f63d8008840 2021-03-09T10:03:48.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:48.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022790 con 0x7f6400146050 2021-03-09T10:03:48.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:48.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:48.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:48.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:48.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.410+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:48.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.411+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:48.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.411+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:03:48.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.411+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:48.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.411+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:48.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.411+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:48.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.411+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:48.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.411+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:48.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:48.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:48.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:48.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:48.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:48.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:03:48.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:03:48.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:48.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:48.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00679d0 con 0x7f6400146050 2021-03-09T10:03:48.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:48.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:48.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f6400146050 2021-03-09T10:03:48.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:03:48.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:48.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12447 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0088f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:48.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:03:48.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.412+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:48.424 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:48.413+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34278 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:48.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.414+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T10:03:48.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.414+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:48.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.414+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:48.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.414+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:48.426 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:48.426 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:48.427 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:48.427 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:48.427 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:48.427 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:48.427 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:48.428 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:48.428 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:48.428 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:48.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.416+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:03:48.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.416+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:48.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.416+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:03:48.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.420+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T10:03:48.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:48.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:48.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:48.666+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:49.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:49.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:49.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:49.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:50.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:50.216+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:50.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:50.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:50.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:50.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:51.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:51.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:51.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:51.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:52.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:52.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:52.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:52.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:53.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:53.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:53.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12447 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0088f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:53.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12447 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0088f90).stop 2021-03-09T10:03:53.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:53.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:53.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:53.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:53.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:03:53.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:53.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:53.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:53.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:53.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:53.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:53.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:53.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:03:53.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:53.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T10:03:53.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:53.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8005e00 2021-03-09T10:03:53.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:53.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:53.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:53.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:53.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.413+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:53.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.414+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:53.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.414+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:03:53.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.414+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:53.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.414+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:53.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.414+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:53.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.414+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:53.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.414+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:53.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:53.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:53.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:53.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:53.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:53.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:03:53.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:03:53.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:53.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:53.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cf50 con 0x7f63d8005e00 2021-03-09T10:03:53.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:53.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:53.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00679d0 con 0x7f63d8005e00 2021-03-09T10:03:53.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:03:53.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.415+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:53.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.416+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023880 secure :-1 s=READY pgs=12458 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0039fc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:53.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.416+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:03:53.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.416+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:53.429 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:53.419+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34318 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:53.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.420+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8005e00 2021-03-09T10:03:53.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.420+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:53.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.420+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:53.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.420+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:53.431 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:53.431 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:53.431 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:53.432 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:53.432 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:53.432 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:53.432 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:53.432 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:53.433 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:53.433 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:53.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.424+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8005e00 2021-03-09T10:03:53.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.424+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:53.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.424+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:03:53.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.424+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004c430 con 0x7f63d8005e00 2021-03-09T10:03:53.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:53.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:53.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:54.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:54.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:54.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:54.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:55.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:55.216+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:03:55.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:55.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:55.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:55.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:56.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:56.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:56.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:56.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:57.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:57.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:57.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:57.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:58.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.419+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0023880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:03:58.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.419+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0023880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:03:58.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.419+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023880 secure :-1 s=READY pgs=12458 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0039fc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:03:58.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023880 secure :-1 s=READY pgs=12458 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0039fc0).stop 2021-03-09T10:03:58.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:58.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:03:58.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0023880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:03:58.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023880 unknown :-1 s=CLOSED pgs=12458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:03:58.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:03:58.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:58.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:03:58.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:58.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:03:58.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:03:58.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:03:58.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:58.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8003400 2021-03-09T10:03:58.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:58.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025930 con 0x7f63d8008840 2021-03-09T10:03:58.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:03:58.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068b10 con 0x7f6400146050 2021-03-09T10:03:58.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:03:58.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:03:58.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:58.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:03:58.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:03:58.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:03:58.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:03:58.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:03:58.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.420+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:03:58.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:03:58.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:03:58.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:03:58.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:03:58.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:58.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e0023390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:58.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:03:58.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:03:58.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:03:58.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:03:58.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:58.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:58.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015f70 con 0x7f6400146050 2021-03-09T10:03:58.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:03:58.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:03:58.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c5d0 con 0x7f6400146050 2021-03-09T10:03:58.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:03:58.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.421+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:58.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.422+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12467 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04bd20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:03:58.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.422+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:03:58.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.422+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:03:58.435 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:03:58.425+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34356 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:03:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc029e30 con 0x7f6400146050 2021-03-09T10:03:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.425+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:03:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.425+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:03:58.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.425+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:03:58.437 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:03:58.437 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:03:58.437 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:58.437 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:03:58.437 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:03:58.438 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:03:58.438 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:03:58.438 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:03:58.438 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:03:58.439 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:03:58.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f6400146050 2021-03-09T10:03:58.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.425+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:03:58.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.425+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:03:58.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T10:03:58.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:58.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:03:58.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:58.666+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:03:59.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:59.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:03:59.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:03:59.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:00.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:00.217+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:00.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:00.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:00.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:00.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:01.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:01.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:01.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:01.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:02.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:02.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:02.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:02.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:03.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.425+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:03.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.425+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:03.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12467 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04bd20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:03.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12467 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc04bd20).stop 2021-03-09T10:04:03.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.425+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:03.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.425+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:03.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:03.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.425+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:03.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.425+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:04:03.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:03.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:03.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:03.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:03.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:03.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:03.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:03.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010500 con 0x7f63d8003400 2021-03-09T10:04:03.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:03.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:04:03.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:03.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8005e00 2021-03-09T10:04:03.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:03.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:03.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:03.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:03.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:03.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.427+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:03.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.427+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:04:03.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.427+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:03.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.427+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:03.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.428+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:03.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.428+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:03.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.428+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:03.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:03.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:03.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:03.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:03.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:03.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:04:03.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:04:03.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:03.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:03.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c5d0 con 0x7f63d8005e00 2021-03-09T10:04:03.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:03.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:03.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cf50 con 0x7f63d8005e00 2021-03-09T10:04:03.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:04:03.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:03.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e100 secure :-1 s=READY pgs=12488 cs=0 l=1 rev1=1 rx=0x7f63f007e070 tx=0x7f63f0006c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:03.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:04:03.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.429+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:03.442 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:03.431+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34396 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:03.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.432+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009580 con 0x7f63d8005e00 2021-03-09T10:04:03.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.432+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:03.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.432+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:03.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.432+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:03.444 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:03.444 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:03.444 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:03.445 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:03.445 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:03.445 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:03.445 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:03.445 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:03.446 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:03.446 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:03.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.436+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0038e80 con 0x7f63d8005e00 2021-03-09T10:04:03.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.436+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:03.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.436+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:04:03.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.436+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:04:03.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:03.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:03.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:04.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:04.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:04.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:04.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:05.218+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:05.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:05.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:05.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:05.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:06.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:06.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:06.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:06.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:07.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:07.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:07.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:07.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:08.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:08.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:08.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e100 secure :-1 s=READY pgs=12488 cs=0 l=1 rev1=1 rx=0x7f63f007e070 tx=0x7f63f0006c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:08.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e100 secure :-1 s=READY pgs=12488 cs=0 l=1 rev1=1 rx=0x7f63f007e070 tx=0x7f63f0006c30).stop 2021-03-09T10:04:08.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:08.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:08.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000e100 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:08.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000e100 unknown :-1 s=CLOSED pgs=12488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:08.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:04:08.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:08.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:08.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:08.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:08.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:08.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:08.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:08.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T10:04:08.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:08.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023020 con 0x7f63d8008840 2021-03-09T10:04:08.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:08.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc60 con 0x7f6400146050 2021-03-09T10:04:08.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.431+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:08.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.432+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:08.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.432+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:08.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.432+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:08.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.432+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:08.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.433+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:08.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.433+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:04:08.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.433+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:08.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.433+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:08.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.433+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:08.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.433+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:08.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.433+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:08.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:08.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:08.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:08.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:08.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:08.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:04:08.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:04:08.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:08.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:08.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005dd10 con 0x7f6400146050 2021-03-09T10:04:08.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:08.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:08.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015f70 con 0x7f6400146050 2021-03-09T10:04:08.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:04:08.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:08.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:04:08.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:08.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.435+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12504 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc02b3b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:08.449 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:08.436+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34434 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:08.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.437+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02b5e0 con 0x7f6400146050 2021-03-09T10:04:08.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.437+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:08.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.437+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:08.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.437+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:08.451 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:08.451 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:08.452 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:08.452 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:08.452 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:08.452 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:08.453 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:08.453 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:08.453 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:08.453 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:08.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.439+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02b5e0 con 0x7f6400146050 2021-03-09T10:04:08.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.439+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:08.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.439+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:04:08.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f6400146050 2021-03-09T10:04:08.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:08.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:08.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:08.668+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:09.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:09.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:09.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:10.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:10.219+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:10.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:10.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:10.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:11.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:11.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:11.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:11.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:12.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:12.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:12.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:12.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:13.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:13.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:13.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12504 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc02b3b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:13.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12504 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc02b3b0).stop 2021-03-09T10:04:13.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:13.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:13.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:13.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:13.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:04:13.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:13.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:13.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:13.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:13.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:13.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:13.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:13.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:04:13.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:13.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b450 con 0x7f63d8008840 2021-03-09T10:04:13.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:13.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068b10 con 0x7f63d8005e00 2021-03-09T10:04:13.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:13.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:13.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:13.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:13.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.437+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:13.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.438+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:13.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.438+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:04:13.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.438+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:13.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.438+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:13.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.439+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:13.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.439+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:13.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.439+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:13.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:13.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:13.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:13.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:13.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:13.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:04:13.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:04:13.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:13.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:13.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028450 con 0x7f63d8005e00 2021-03-09T10:04:13.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:13.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:13.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005dd10 con 0x7f63d8005e00 2021-03-09T10:04:13.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:04:13.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:13.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.441+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:04:13.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.441+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:13.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.440+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 secure :-1 s=READY pgs=12514 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc066cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:13.453 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:13.442+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34473 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:13.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T10:04:13.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.442+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:13.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.442+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:13.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.442+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:13.455 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:13.455 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:13.455 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:13.456 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:13.456 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:13.456 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:13.456 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:13.456 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:13.457 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:13.457 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:13.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T10:04:13.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.443+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:13.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.443+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:04:13.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.444+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0125b0 con 0x7f63d8005e00 2021-03-09T10:04:13.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:13.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:13.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:14.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:14.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:14.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:14.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:15.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:15.220+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:15.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:15.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:15.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:15.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:16.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:16.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:16.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:16.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:17.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:17.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:17.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:17.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00299c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00299c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:18.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 secure :-1 s=READY pgs=12514 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc066cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:18.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 secure :-1 s=READY pgs=12514 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc066cf0).stop 2021-03-09T10:04:18.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:18.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:18.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00299c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:18.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 unknown :-1 s=CLOSED pgs=12514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:18.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:04:18.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:18.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:18.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:18.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:18.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:18.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:18.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:18.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:04:18.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:18.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a750 con 0x7f63d8008840 2021-03-09T10:04:18.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:18.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00679d0 con 0x7f6400146050 2021-03-09T10:04:18.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.442+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:18.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.443+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:18.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.443+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:18.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.443+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:18.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.443+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:18.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.443+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:18.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.443+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:04:18.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.443+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:18.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.443+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:18.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:18.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:18.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:18.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:18.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:18.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:18.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:18.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:18.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:04:18.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:04:18.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:18.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:18.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b970 con 0x7f6400146050 2021-03-09T10:04:18.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:18.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:18.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028450 con 0x7f6400146050 2021-03-09T10:04:18.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:04:18.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.444+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:18.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.445+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:04:18.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.445+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:18.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.445+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12523 cs=0 l=1 rev1=1 rx=0x7f63e005dea0 tx=0x7f63fc07afb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:18.460 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:18.448+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34512 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:18.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.449+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f6400146050 2021-03-09T10:04:18.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.449+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:18.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.449+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:18.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.449+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:18.462 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:18.462 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:18.462 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:18.463 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:18.463 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:18.463 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:18.463 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:18.464 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:18.464 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:18.464 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:18.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f6400146050 2021-03-09T10:04:18.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.450+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:18.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.450+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:04:18.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc07a330 con 0x7f6400146050 2021-03-09T10:04:18.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:18.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:18.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:18.669+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:19.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:19.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:19.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:19.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:20.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:20.221+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:20.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:20.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:20.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:21.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:21.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:21.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:21.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:22.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:22.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:22.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:22.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:23.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:23.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:23.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12523 cs=0 l=1 rev1=1 rx=0x7f63e005dea0 tx=0x7f63fc07afb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:23.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12523 cs=0 l=1 rev1=1 rx=0x7f63e005dea0 tx=0x7f63fc07afb0).stop 2021-03-09T10:04:23.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:23.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:23.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:23.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:23.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:04:23.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:23.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:23.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:23.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:23.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:23.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:23.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:23.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e090 con 0x7f63d8003400 2021-03-09T10:04:23.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:23.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064ef0 con 0x7f63d8008840 2021-03-09T10:04:23.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:23.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bbc0 con 0x7f63d8005e00 2021-03-09T10:04:23.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:23.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:23.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:23.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:23.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.449+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:23.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.450+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:23.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.450+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:04:23.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.450+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:23.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.450+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:23.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.450+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:23.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.450+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:23.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.450+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:23.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:23.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:23.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:23.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:04:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:04:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028450 con 0x7f63d8005e00 2021-03-09T10:04:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c5d0 con 0x7f63d8005e00 2021-03-09T10:04:23.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:04:23.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.451+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:23.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.452+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:04:23.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.452+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:23.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.452+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 secure :-1 s=READY pgs=12541 cs=0 l=1 rev1=1 rx=0x7f63e001c0d0 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:23.464 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:23.455+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34550 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:23.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00412d0 con 0x7f63d8005e00 2021-03-09T10:04:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.455+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.455+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.455+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:23.465 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:23.466 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:23.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:04:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.458+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.458+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:04:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:04:23.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:23.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:23.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:24.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:24.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:24.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:24.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:25.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:25.222+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:25.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:25.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:25.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:26.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:26.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:26.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:26.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:27.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:27.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:27.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:27.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:28.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.455+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00299c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:28.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.455+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00299c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:28.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.455+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 secure :-1 s=READY pgs=12541 cs=0 l=1 rev1=1 rx=0x7f63e001c0d0 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:28.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.455+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 secure :-1 s=READY pgs=12541 cs=0 l=1 rev1=1 rx=0x7f63e001c0d0 tx=0x7f63f0000c30).stop 2021-03-09T10:04:28.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.455+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:28.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.455+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:28.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00299c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:28.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299c0 unknown :-1 s=CLOSED pgs=12541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:28.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:04:28.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:28.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:28.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:28.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:28.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:28.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:28.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:28.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00280e0 con 0x7f63d8003400 2021-03-09T10:04:28.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:28.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8008840 2021-03-09T10:04:28.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:28.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:04:28.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:28.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:28.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:28.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:28.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:28.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:28.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:04:28.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:28.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.456+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:28.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:28.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:28.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:28.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:28.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:04:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:04:28.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005dd40 con 0x7f6400146050 2021-03-09T10:04:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028450 con 0x7f6400146050 2021-03-09T10:04:28.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:04:28.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:28.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:04:28.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12550 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004cf20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:28.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.457+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:28.471 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:28.460+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34590 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005ff00 con 0x7f6400146050 2021-03-09T10:04:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.461+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:28.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.461+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.461+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:28.473 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:28.474 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:28.474 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:28.474 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:28.474 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:28.475 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:28.475 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:28.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T10:04:28.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.461+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:28.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.461+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:04:28.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.462+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f6400146050 2021-03-09T10:04:28.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:28.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:28.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:28.670+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:29.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:29.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:29.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:29.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:30.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:30.223+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:30.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:30.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:30.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:30.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:31.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:31.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:31.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:31.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:32.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:32.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:32.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:32.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:33.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12550 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004cf20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:33.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12550 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004cf20).stop 2021-03-09T10:04:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:33.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:04:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:33.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027fa0 con 0x7f63d8003400 2021-03-09T10:04:33.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:33.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010500 con 0x7f63d8008840 2021-03-09T10:04:33.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:33.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266f0 con 0x7f63d8005e00 2021-03-09T10:04:33.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:33.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:33.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:33.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:33.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:04:33.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:33.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:33.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:33.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:33.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:33.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:33.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:33.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:33.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:33.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:33.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:33.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:33.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:04:33.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:04:33.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:33.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:33.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e500 con 0x7f63d8005e00 2021-03-09T10:04:33.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:33.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:33.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005dd40 con 0x7f63d8005e00 2021-03-09T10:04:33.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:04:33.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.462+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:33.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.463+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:04:33.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.463+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299f0 secure :-1 s=READY pgs=12563 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004cda0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:33.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.463+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:33.476 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:33.466+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34626 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:33.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.466+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00556a0 con 0x7f63d8005e00 2021-03-09T10:04:33.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.466+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:33.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.466+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:33.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.466+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:33.478 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:33.478 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:33.478 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:33.479 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:33.479 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:33.479 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:33.479 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:33.480 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:33.480 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:33.480 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:33.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:04:33.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.467+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:33.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.467+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:04:33.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055d70 con 0x7f63d8005e00 2021-03-09T10:04:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:33.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:33.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:34.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:34.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:34.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:34.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:35.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:35.224+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:35.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:35.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:35.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:35.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:36.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:36.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:36.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:36.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:37.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:37.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:37.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:37.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:38.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00299f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:38.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00299f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:38.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299f0 secure :-1 s=READY pgs=12563 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004cda0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:38.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299f0 secure :-1 s=READY pgs=12563 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004cda0).stop 2021-03-09T10:04:38.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:38.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:38.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00299f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:38.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00299f0 unknown :-1 s=CLOSED pgs=12563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:38.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:04:38.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:38.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:38.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:38.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:38.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:38.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:38.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:38.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022790 con 0x7f63d8003400 2021-03-09T10:04:38.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:38.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8008840 2021-03-09T10:04:38.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:38.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066a10 con 0x7f6400146050 2021-03-09T10:04:38.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:38.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:38.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:38.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:38.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.467+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:38.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.468+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:38.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.468+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:04:38.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.468+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:38.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.468+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:38.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.468+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:38.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.468+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:38.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.468+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:38.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:38.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:38.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:38.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:38.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:38.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:04:38.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:04:38.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:38.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:38.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T10:04:38.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:38.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:38.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b970 con 0x7f6400146050 2021-03-09T10:04:38.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:04:38.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.469+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:38.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.470+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12582 cs=0 l=1 rev1=1 rx=0x7f63e005e500 tx=0x7f63fc008cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.470+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:04:38.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.470+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:38.482 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:38.472+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34668 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:38.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f6400146050 2021-03-09T10:04:38.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.473+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:38.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.473+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:38.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.473+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:38.484 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:38.484 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:38.484 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:38.485 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:38.485 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:38.485 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:38.485 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:38.485 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:38.486 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:38.486 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:38.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f6400146050 2021-03-09T10:04:38.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.473+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:38.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.474+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:04:38.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.475+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T10:04:38.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:38.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:38.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:38.671+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:39.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:39.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:39.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:39.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:40.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:40.225+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:40.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:40.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:40.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:40.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:41.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:41.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:41.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:41.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:42.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:42.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:42.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:42.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:43.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.472+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:04:43.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.472+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:43.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.472+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12582 cs=0 l=1 rev1=1 rx=0x7f63e005e500 tx=0x7f63fc008cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:43.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.472+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12582 cs=0 l=1 rev1=1 rx=0x7f63e005e500 tx=0x7f63fc008cf0).stop 2021-03-09T10:04:43.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:43.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:04:43.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:43.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:43.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:43.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:43.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:43.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:43.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:43.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266f0 con 0x7f63d8003400 2021-03-09T10:04:43.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:43.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8008840 2021-03-09T10:04:43.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:43.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:04:43.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:43.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:43.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:43.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:43.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.473+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:43.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.474+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:43.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.474+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:04:43.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.474+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:43.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.474+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:43.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.474+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:43.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.474+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:43.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.474+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:43.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:43.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:43.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:43.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:43.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:43.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:04:43.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:04:43.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:43.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:43.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c5d0 con 0x7f63d8005e00 2021-03-09T10:04:43.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:43.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:43.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e500 con 0x7f63d8005e00 2021-03-09T10:04:43.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:04:43.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.475+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:43.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.476+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=12594 cs=0 l=1 rev1=1 rx=0x7f63f004cef0 tx=0x7f63f0038f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:43.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.476+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:04:43.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.476+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:43.489 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:43.478+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34706 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:43.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.478+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8005e00 2021-03-09T10:04:43.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.478+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:43.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.478+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:43.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.478+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:43.491 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:43.491 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:43.491 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:43.491 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:43.491 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:43.492 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:43.492 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:43.492 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:43.492 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:43.492 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:43.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.481+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:04:43.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.481+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:43.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.481+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:04:43.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.482+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:04:43.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:43.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:43.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:44.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:44.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:44.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:44.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:45.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:45.226+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:45.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:45.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:45.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:45.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:46.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:46.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:46.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:47.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:47.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:47.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:48.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:04:48.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:48.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=12594 cs=0 l=1 rev1=1 rx=0x7f63f004cef0 tx=0x7f63f0038f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:48.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=12594 cs=0 l=1 rev1=1 rx=0x7f63f004cef0 tx=0x7f63f0038f60).stop 2021-03-09T10:04:48.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:48.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:48.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:48.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=CLOSED pgs=12594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:48.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:04:48.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:48.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:48.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:48.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:48.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:48.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:48.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:48.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8003400 2021-03-09T10:04:48.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:48.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010500 con 0x7f63d8008840 2021-03-09T10:04:48.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:48.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c580 con 0x7f6400146050 2021-03-09T10:04:48.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.478+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.479+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.479+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.479+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:48.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.480+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:04:48.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.480+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:48.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.480+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:48.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.480+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:48.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.480+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:48.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.480+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:48.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:48.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:48.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:48.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:48.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:48.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:04:48.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:04:48.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:48.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:48.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f240 con 0x7f6400146050 2021-03-09T10:04:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c5d0 con 0x7f6400146050 2021-03-09T10:04:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:04:48.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.481+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.482+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12612 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0055290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.482+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:04:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.482+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:48.494 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:48.484+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34746 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040ec0 con 0x7f6400146050 2021-03-09T10:04:48.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.484+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:48.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.484+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:48.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.484+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:48.496 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:48.496 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:48.496 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:48.497 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:48.497 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:48.497 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:48.497 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:48.498 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:48.498 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:48.498 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:48.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.486+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:04:48.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.486+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:48.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.486+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:04:48.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055d70 con 0x7f6400146050 2021-03-09T10:04:48.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:48.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:48.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:48.673+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:49.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:49.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:49.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:49.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:50.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:50.227+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:50.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:50.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:50.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:50.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:51.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:51.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:51.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:51.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:52.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:52.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:52.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:52.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:53.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.484+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:53.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.484+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:53.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.484+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12612 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0055290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:53.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.484+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12612 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f0055290).stop 2021-03-09T10:04:53.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:53.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:53.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:53.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:53.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:04:53.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:53.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:53.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:53.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:53.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:53.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:53.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:53.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8003400 2021-03-09T10:04:53.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:53.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f100 con 0x7f63d8008840 2021-03-09T10:04:53.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:53.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:04:53.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:53.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:53.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:53.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:53.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.485+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:53.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.486+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:53.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.486+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:04:53.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.486+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:53.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.486+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:53.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.486+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:53.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.486+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:53.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.486+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:53.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:53.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:53.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:53.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:53.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:53.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:04:53.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:04:53.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:53.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:53.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b70 con 0x7f63d8005e00 2021-03-09T10:04:53.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:53.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:53.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:04:53.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:04:53.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:53.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.487+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=12621 cs=0 l=1 rev1=1 rx=0x7f63fc071360 tx=0x7f63fc065900).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:53.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.488+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:04:53.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.488+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:53.507 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:53.490+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34784 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:53.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T10:04:53.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.491+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:53.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.491+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:53.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.491+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:53.509 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:53.510 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:53.510 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:53.510 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:53.511 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:53.511 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:53.511 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:53.512 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:53.512 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:53.512 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:53.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.493+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T10:04:53.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.493+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:53.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.493+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:04:53.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:04:53.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:53.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:53.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:54.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:54.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:54.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:54.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:55.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:55.228+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:04:55.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:55.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:55.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:55.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:56.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:56.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:57.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:57.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:57.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:57.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:58.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:04:58.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:04:58.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=12621 cs=0 l=1 rev1=1 rx=0x7f63fc071360 tx=0x7f63fc065900).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:04:58.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=12621 cs=0 l=1 rev1=1 rx=0x7f63fc071360 tx=0x7f63fc065900).stop 2021-03-09T10:04:58.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:58.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:04:58.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:04:58.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=CLOSED pgs=12621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:04:58.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:04:58.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:58.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:04:58.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:58.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:04:58.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:04:58.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:04:58.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:58.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8003400 2021-03-09T10:04:58.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:58.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8008840 2021-03-09T10:04:58.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:04:58.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T10:04:58.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:04:58.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:04:58.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:58.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.491+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:04:58.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.491+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:04:58.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.491+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:04:58.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.491+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:04:58.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.491+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:04:58.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.491+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:04:58.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.491+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:04:58.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.491+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:04:58.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.491+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:04:58.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:04:58.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:58.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:58.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:04:58.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:04:58.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:04:58.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:04:58.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:58.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:58.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e500 con 0x7f6400146050 2021-03-09T10:04:58.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:04:58.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:04:58.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b70 con 0x7f6400146050 2021-03-09T10:04:58.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:04:58.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.492+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:58.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.493+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:04:58.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.493+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:04:58.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.493+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12635 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc066e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:04:58.506 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:04:58.496+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34824 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:04:58.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06d3a0 con 0x7f6400146050 2021-03-09T10:04:58.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.496+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:04:58.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.496+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:04:58.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.496+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:04:58.508 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:04:58.508 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:04:58.509 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:58.509 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:04:58.509 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:04:58.509 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:04:58.509 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:04:58.510 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:04:58.510 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:04:58.510 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:04:58.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.501+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06d3a0 con 0x7f6400146050 2021-03-09T10:04:58.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.501+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:04:58.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.501+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f6400146050 2021-03-09T10:04:58.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.502+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:04:58.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:58.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:04:58.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:58.674+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:04:59.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:59.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:04:59.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:04:59.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:00.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:00.229+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:00.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:00.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:00.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:00.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:01.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:01.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:01.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:01.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:02.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:02.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:02.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:02.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:03.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12635 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc066e40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12635 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc066e40).stop 2021-03-09T10:05:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:03.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:05:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:03.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:03.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:05:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028860 con 0x7f63d8008840 2021-03-09T10:05:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e00 con 0x7f63d8005e00 2021-03-09T10:05:03.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:03.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:03.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.496+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:03.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.497+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:03.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.497+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:03.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.497+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:03.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.497+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:05:03.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.497+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:03.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.497+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:03.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:03.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:03.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:03.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:03.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:03.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:03.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:03.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:03.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:05:03.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:05:03.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:03.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:03.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d430 con 0x7f63d8005e00 2021-03-09T10:05:03.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:03.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:03.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f240 con 0x7f63d8005e00 2021-03-09T10:05:03.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:05:03.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.498+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:03.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.499+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=12648 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0015920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:03.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.499+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:05:03.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.499+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:03.512 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:03.502+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34862 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:03.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8005e00 2021-03-09T10:05:03.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.502+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:03.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.502+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:03.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.502+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:03.514 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:03.514 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:03.514 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:03.515 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:03.515 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:03.515 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:03.515 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:03.515 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:03.516 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:03.516 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:03.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.505+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027e70 con 0x7f63d8005e00 2021-03-09T10:05:03.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.505+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:03.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.506+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:05:03.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:05:03.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:03.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:03.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:04.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:04.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:04.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:04.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:05.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:05.230+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:05.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:05.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:05.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:05.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:06.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:06.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:06.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:06.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:07.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:07.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:07.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:07.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:08.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:08.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:08.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=12648 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0015920).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:08.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=12648 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0015920).stop 2021-03-09T10:05:08.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:08.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:08.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:08.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=CLOSED pgs=12648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:08.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:05:08.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:08.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:08.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:08.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:08.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:05:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:08.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065a40 con 0x7f63d8008840 2021-03-09T10:05:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f6400146050 2021-03-09T10:05:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.502+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:08.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.503+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:08.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.503+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:08.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.503+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:08.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.503+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:08.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.504+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:08.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.504+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:05:08.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.504+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:08.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.504+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:08.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.504+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:08.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.504+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:08.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.504+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:08.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.504+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:08.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:08.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:08.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:08.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:08.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:05:08.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:05:08.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:08.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:08.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005ec50 con 0x7f6400146050 2021-03-09T10:05:08.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:08.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:08.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d430 con 0x7f6400146050 2021-03-09T10:05:08.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:05:08.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.505+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:08.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12666 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f007fdf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:08.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:05:08.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:08.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:08.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:08.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12666 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f007fdf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:08.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12666 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f007fdf0).stop 2021-03-09T10:05:08.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:08.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:08.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:08.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:08.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:05:08.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:08.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:08.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:08.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:08.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:08.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:08.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:08.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023c70 con 0x7f63d8003400 2021-03-09T10:05:08.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:08.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00679d0 con 0x7f63d8008840 2021-03-09T10:05:08.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:08.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017860 con 0x7f63d8005e00 2021-03-09T10:05:08.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:08.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:08.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.506+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:08.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.507+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:08.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.507+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:08.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.507+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:08.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.507+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:05:08.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.507+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:08.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.507+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:08.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.507+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:08.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.507+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:08.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.507+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:08.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:08.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:08.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:08.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:08.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:08.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:05:08.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:05:08.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:08.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:08.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d430 con 0x7f63d8005e00 2021-03-09T10:05:08.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:08.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:08.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005ec50 con 0x7f63d8005e00 2021-03-09T10:05:08.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:05:08.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:08.532 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:08.508+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34902 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:08.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.509+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:05:08.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.509+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 secure :-1 s=READY pgs=12672 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f0006c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:08.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.509+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:08.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.515+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009580 con 0x7f63d8005e00 2021-03-09T10:05:08.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.515+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:08.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.515+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:08.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.515+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:08.534 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:08.535 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:08.535 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:08.535 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:08.535 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:08.536 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:08.536 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:08.536 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:08.536 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:08.536 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:08.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.515+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00284b0 con 0x7f63d8005e00 2021-03-09T10:05:08.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.515+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:08.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.515+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:05:08.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.515+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0040ad0 con 0x7f63d8005e00 2021-03-09T10:05:08.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:08.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:08.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:08.676+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:09.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:09.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:09.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:09.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:10.231+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:10.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:10.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:10.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:10.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:11.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:11.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:11.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:11.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:12.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:12.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:12.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:12.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:13.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:05:13.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:13.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 secure :-1 s=READY pgs=12672 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f0006c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:13.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 secure :-1 s=READY pgs=12672 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f0006c20).stop 2021-03-09T10:05:13.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:13.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:13.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:13.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 unknown :-1 s=CLOSED pgs=12672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:13.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:05:13.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:13.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:13.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:13.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:13.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:13.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:13.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:13.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001950 con 0x7f63d8003400 2021-03-09T10:05:13.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:13.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63d8008840 2021-03-09T10:05:13.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:13.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:05:13.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:13.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:13.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.509+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:13.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.510+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:13.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.510+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:13.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.510+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:13.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.510+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:05:13.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.510+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:13.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.510+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:13.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.510+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:13.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.510+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:13.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.510+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:13.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:13.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:13.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:13.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:13.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:13.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:05:13.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:05:13.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:13.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:13.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019c10 con 0x7f6400146050 2021-03-09T10:05:13.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:13.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:13.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d430 con 0x7f6400146050 2021-03-09T10:05:13.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:05:13.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:13.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:05:13.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.511+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12679 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0009580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:13.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.512+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:13.525 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:13.514+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34940 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:13.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.515+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00284b0 con 0x7f6400146050 2021-03-09T10:05:13.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.515+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:13.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.515+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:13.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.515+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:13.526 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:13.526 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:13.527 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:13.527 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:13.527 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:13.527 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:13.528 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:13.528 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:13.528 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:13.528 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:13.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005fec0 con 0x7f6400146050 2021-03-09T10:05:13.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.519+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:13.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f6400146050 2021-03-09T10:05:13.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.520+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:05:13.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:13.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:13.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:14.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:14.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:14.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:14.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:15.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:15.232+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:15.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:15.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:15.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:15.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:16.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:16.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:16.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:16.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:17.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:17.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:17.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:17.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:18.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:18.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:18.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12679 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0009580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:18.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12679 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0009580).stop 2021-03-09T10:05:18.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:18.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:18.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:18.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:05:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:18.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017860 con 0x7f63d8003400 2021-03-09T10:05:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:05:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010500 con 0x7f63d8005e00 2021-03-09T10:05:18.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.514+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:18.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:18.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:18.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:18.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:18.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:18.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:05:18.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:18.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:18.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:18.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:18.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.515+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:18.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:18.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:18.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:18.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:18.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:18.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:05:18.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:05:18.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:18.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:18.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cf50 con 0x7f63d8005e00 2021-03-09T10:05:18.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:18.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:18.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019c10 con 0x7f63d8005e00 2021-03-09T10:05:18.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:05:18.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:18.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:05:18.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b2c0 secure :-1 s=READY pgs=12687 cs=0 l=1 rev1=1 rx=0x7f63f00159f0 tx=0x7f63f0055510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:18.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.516+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:18.530 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:18.520+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 34980 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:18.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.521+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:05:18.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.521+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:18.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.521+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:18.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.521+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:18.532 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:18.532 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:18.532 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:18.533 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:18.533 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:18.533 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:18.533 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:18.534 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:18.534 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:18.534 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:18.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.523+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007ae00 con 0x7f63d8005e00 2021-03-09T10:05:18.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.523+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:18.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.523+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:05:18.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00112f0 con 0x7f63d8005e00 2021-03-09T10:05:18.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:18.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:18.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:18.677+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:19.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:19.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:19.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:20.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:20.233+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:20.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:20.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:20.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:20.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:21.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:21.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:21.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:21.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:22.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:22.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:22.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:22.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:23.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:23.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b2c0 secure :-1 s=READY pgs=12687 cs=0 l=1 rev1=1 rx=0x7f63f00159f0 tx=0x7f63f0055510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:23.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b2c0 secure :-1 s=READY pgs=12687 cs=0 l=1 rev1=1 rx=0x7f63f00159f0 tx=0x7f63f0055510).stop 2021-03-09T10:05:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:23.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b2c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b2c0 unknown :-1 s=CLOSED pgs=12687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:23.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:05:23.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:23.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:23.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:23.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:23.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:23.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:23.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:23.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016f20 con 0x7f63d8003400 2021-03-09T10:05:23.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:23.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022790 con 0x7f63d8008840 2021-03-09T10:05:23.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:23.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d760 con 0x7f6400146050 2021-03-09T10:05:23.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:23.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:23.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:23.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.520+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:23.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:05:23.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:23.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:23.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:23.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:23.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:23.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:23.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:23.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:23.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:23.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:23.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:05:23.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:05:23.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:23.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:23.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005ec50 con 0x7f6400146050 2021-03-09T10:05:23.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:23.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:23.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cf50 con 0x7f6400146050 2021-03-09T10:05:23.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:05:23.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.521+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:23.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.522+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12702 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0073420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:23.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.522+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:05:23.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.522+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:23.547 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:23.526+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35018 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:23.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.526+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f6400146050 2021-03-09T10:05:23.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.526+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:23.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.526+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:23.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.526+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:23.550 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:23.550 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:23.551 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:23.551 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:23.552 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:23.552 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:23.552 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:23.553 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:23.553 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:23.554 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:23.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.526+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f6400146050 2021-03-09T10:05:23.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.526+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:23.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.526+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:05:23.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.527+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T10:05:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:23.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:24.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:24.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:24.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:24.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:25.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:25.234+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:25.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:25.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:25.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:25.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:26.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:26.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:26.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:26.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:27.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:27.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:27.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:27.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:28.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:28.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:28.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12702 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0073420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:28.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12702 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0073420).stop 2021-03-09T10:05:28.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:28.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:28.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:28.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:28.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:05:28.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:28.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:28.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:28.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:28.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0024aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:28.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:28.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:28.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:05:28.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:28.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T10:05:28.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:28.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007be10 con 0x7f63d8005e00 2021-03-09T10:05:28.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:28.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:28.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:28.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:28.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:28.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0024aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:28.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:05:28.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:28.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:28.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:28.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:28.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.527+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:28.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:28.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:28.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:28.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:28.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:28.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:05:28.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:05:28.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:28.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:28.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T10:05:28.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:28.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:28.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005ec50 con 0x7f63d8005e00 2021-03-09T10:05:28.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:05:28.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:28.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:05:28.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:28.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.528+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0024aa0 secure :-1 s=READY pgs=12716 cs=0 l=1 rev1=1 rx=0x7f63f00159f0 tx=0x7f63f0087410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:28.541 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:28.531+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35058 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:28.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.532+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:05:28.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.532+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:28.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.532+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:28.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.532+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:28.543 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:28.543 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:28.543 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:28.544 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:28.544 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:28.544 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:28.544 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:28.544 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:28.545 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:28.545 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:28.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.532+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:05:28.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.532+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:28.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.532+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:05:28.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.533+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055b80 con 0x7f63d8005e00 2021-03-09T10:05:28.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:28.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:28.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:28.678+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:29.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:29.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:29.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:29.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:30.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:30.235+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:30.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:30.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:30.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:30.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:31.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:31.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:31.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:31.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:32.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:32.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:32.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:32.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:33.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0024aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:33.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0024aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:33.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0024aa0 secure :-1 s=READY pgs=12716 cs=0 l=1 rev1=1 rx=0x7f63f00159f0 tx=0x7f63f0087410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:33.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0024aa0 secure :-1 s=READY pgs=12716 cs=0 l=1 rev1=1 rx=0x7f63f00159f0 tx=0x7f63f0087410).stop 2021-03-09T10:05:33.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:33.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:33.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0024aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:33.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0024aa0 unknown :-1 s=CLOSED pgs=12716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:33.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:05:33.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:33.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:33.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:33.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:33.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:33.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:33.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:33.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:05:33.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:33.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005eca0 con 0x7f63d8008840 2021-03-09T10:05:33.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:33.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d760 con 0x7f6400146050 2021-03-09T10:05:33.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:33.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:33.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:33.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:33.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.532+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:33.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.533+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:33.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.533+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:05:33.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.533+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:33.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.533+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:33.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.533+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:33.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.533+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:33.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.533+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:33.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.534+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:33.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:33.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:33.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:33.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:33.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:05:33.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:05:33.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:33.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:33.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T10:05:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e500 con 0x7f6400146050 2021-03-09T10:05:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:05:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12735 cs=0 l=1 rev1=1 rx=0x7f63e001cdb0 tx=0x7f63fc01b1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:05:33.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.535+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:33.548 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:33.538+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35096 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:33.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f6400146050 2021-03-09T10:05:33.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.538+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:33.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.538+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:33.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.538+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:33.549 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:33.550 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:33.550 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:33.550 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:33.550 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:33.551 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:33.551 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:33.551 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:33.551 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:33.551 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f6400146050 2021-03-09T10:05:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.543+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.544+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:05:33.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.545+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f6400146050 2021-03-09T10:05:33.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:33.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:33.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:34.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:34.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:34.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:34.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:35.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:35.236+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:35.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:35.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:35.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:35.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:36.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:36.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:36.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:36.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:37.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:37.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:37.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:37.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:38.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:38.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:38.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12735 cs=0 l=1 rev1=1 rx=0x7f63e001cdb0 tx=0x7f63fc01b1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:38.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12735 cs=0 l=1 rev1=1 rx=0x7f63e001cdb0 tx=0x7f63fc01b1d0).stop 2021-03-09T10:05:38.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:38.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:38.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:38.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:38.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:05:38.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:38.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:38.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:38.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:38.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:38.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:38.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:38.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013250 con 0x7f63d8003400 2021-03-09T10:05:38.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:38.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028860 con 0x7f63d8008840 2021-03-09T10:05:38.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:38.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:05:38.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:38.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:38.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:38.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:05:38.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:38.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.539+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:38.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:38.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:38.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:38.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:38.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:38.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:38.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:38.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:38.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:05:38.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:05:38.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:38.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:38.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028db0 con 0x7f63d8005e00 2021-03-09T10:05:38.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:38.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:38.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:05:38.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:05:38.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:38.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=12744 cs=0 l=1 rev1=1 rx=0x7f63fc0658d0 tx=0x7f63fc051e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:38.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:05:38.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.540+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:38.555 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:38.544+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35136 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:38.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.544+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63d8005e00 2021-03-09T10:05:38.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.544+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:38.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.544+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:38.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.544+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:38.557 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:38.557 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:38.558 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:38.558 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:38.558 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:38.558 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:38.559 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:38.559 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:38.559 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:38.559 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:38.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.544+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05b170 con 0x7f63d8005e00 2021-03-09T10:05:38.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.544+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:38.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.544+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:05:38.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63d8005e00 2021-03-09T10:05:38.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:38.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:38.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:38.680+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:39.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:39.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:39.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:39.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:40.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:40.237+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:40.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:40.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:40.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:40.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:41.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:41.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:41.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:41.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:42.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:42.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:42.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:42.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:43.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:43.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:43.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=12744 cs=0 l=1 rev1=1 rx=0x7f63fc0658d0 tx=0x7f63fc051e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:43.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=12744 cs=0 l=1 rev1=1 rx=0x7f63fc0658d0 tx=0x7f63fc051e90).stop 2021-03-09T10:05:43.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:43.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:43.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:43.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=CLOSED pgs=12744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:43.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:05:43.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:43.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:43.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:43.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:43.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:43.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:43.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:43.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d760 con 0x7f63d8003400 2021-03-09T10:05:43.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:43.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017860 con 0x7f63d8008840 2021-03-09T10:05:43.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:43.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:05:43.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:43.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:43.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:43.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:43.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:43.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:43.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:05:43.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:43.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:43.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:43.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:43.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.545+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:43.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:43.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:43.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:43.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:43.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:43.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:05:43.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:05:43.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:43.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:43.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f6400146050 2021-03-09T10:05:43.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:43.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:43.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f6400146050 2021-03-09T10:05:43.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:05:43.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.546+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:43.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.547+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12758 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00095f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:43.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.547+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:05:43.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.547+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:43.571 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:43.551+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35174 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:43.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f6400146050 2021-03-09T10:05:43.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.551+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:43.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.551+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:43.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.551+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:43.574 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:43.574 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:43.575 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:43.575 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:43.575 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:43.576 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:43.576 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:43.577 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:43.577 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:43.578 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:43.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f6400146050 2021-03-09T10:05:43.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.554+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:43.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.554+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:05:43.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f6400146050 2021-03-09T10:05:43.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:43.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:43.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:44.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:44.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:44.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:44.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:45.238+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:45.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:45.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:45.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:45.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:46.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:46.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:46.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:46.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:47.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:47.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:47.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:47.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:48.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:05:48.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:48.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12758 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00095f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:48.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12758 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00095f0).stop 2021-03-09T10:05:48.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:48.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:48.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:48.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:48.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:05:48.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:48.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:48.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:48.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:48.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:48.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:48.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:48.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022790 con 0x7f63d8003400 2021-03-09T10:05:48.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:48.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013250 con 0x7f63d8008840 2021-03-09T10:05:48.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:48.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:05:48.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:48.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:48.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:48.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:48.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.551+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:48.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:48.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:05:48.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:48.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:48.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:48.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:48.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:48.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:48.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:48.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:48.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:48.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:48.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:05:48.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:05:48.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:48.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:48.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:05:48.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:48.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:48.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028db0 con 0x7f63d8005e00 2021-03-09T10:05:48.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:05:48.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.552+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:48.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.553+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685a0 secure :-1 s=READY pgs=12771 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:48.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.553+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:05:48.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.553+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:48.567 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:48.556+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35214 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:48.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06dd90 con 0x7f63d8005e00 2021-03-09T10:05:48.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.556+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:48.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.556+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:48.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.556+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:48.569 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:48.569 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:48.569 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:48.569 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:48.570 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:48.570 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:48.570 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:48.571 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:48.571 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:48.571 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:48.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06dd90 con 0x7f63d8005e00 2021-03-09T10:05:48.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.556+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:48.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.556+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:05:48.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.558+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:05:48.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:48.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:48.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:48.681+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:49.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:49.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:49.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:49.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:50.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:50.239+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:50.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:50.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:50.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:50.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:51.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:51.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:51.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:51.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:52.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:52.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:52.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:52.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:53.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00685a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:53.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00685a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:53.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685a0 secure :-1 s=READY pgs=12771 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:53.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685a0 secure :-1 s=READY pgs=12771 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a6d0).stop 2021-03-09T10:05:53.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:53.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:53.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00685a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:53.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685a0 unknown :-1 s=CLOSED pgs=12771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:53.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:05:53.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:53.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:53.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:53.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:53.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:53.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:53.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:53.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:05:53.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:53.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028860 con 0x7f63d8008840 2021-03-09T10:05:53.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:53.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:05:53.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:53.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:53.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:53.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:53.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:53.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:53.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:05:53.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:53.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.557+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:53.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.558+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:53.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.558+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:53.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.558+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:53.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:53.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:53.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:53.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:53.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:53.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:05:53.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:05:53.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:53.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:53.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00104b0 con 0x7f6400146050 2021-03-09T10:05:53.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:53.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:53.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T10:05:53.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:05:53.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.559+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:53.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.560+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:05:53.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.560+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:53.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.560+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12790 cs=0 l=1 rev1=1 rx=0x7f63fc012e60 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:53.574 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:53.563+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35252 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:53.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.563+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066eb0 con 0x7f6400146050 2021-03-09T10:05:53.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.563+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:53.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.563+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:53.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.563+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:53.576 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:53.576 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:53.576 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:53.576 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:53.577 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:53.577 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:53.577 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:53.577 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:53.578 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:53.578 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:53.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.565+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc066eb0 con 0x7f6400146050 2021-03-09T10:05:53.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.565+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:53.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.565+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:05:53.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.565+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc008580 con 0x7f6400146050 2021-03-09T10:05:53.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:53.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:53.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:54.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:54.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:54.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:54.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:55.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:55.240+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:05:55.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:55.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:55.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:55.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:56.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:56.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:56.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:56.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:57.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:57.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:57.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:57.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:58.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:05:58.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:05:58.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12790 cs=0 l=1 rev1=1 rx=0x7f63fc012e60 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:05:58.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12790 cs=0 l=1 rev1=1 rx=0x7f63fc012e60 tx=0x7f63fc009990).stop 2021-03-09T10:05:58.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:58.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:05:58.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:05:58.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:05:58.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:05:58.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:58.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:05:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:05:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:05:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:05:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:58.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c540 con 0x7f63d8003400 2021-03-09T10:05:58.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:58.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c900 con 0x7f63d8008840 2021-03-09T10:05:58.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:05:58.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b700 con 0x7f63d8005e00 2021-03-09T10:05:58.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:05:58.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:05:58.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:58.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:05:58.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:05:58.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:05:58.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:05:58.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:05:58.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.564+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:05:58.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.565+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:05:58.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.565+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:05:58.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.565+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:05:58.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.566+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:05:58.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:58.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:58.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:05:58.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:05:58.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:05:58.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:05:58.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:58.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:58.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:05:58.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:05:58.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:05:58.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63d8005e00 2021-03-09T10:05:58.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:05:58.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:58.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=12807 cs=0 l=1 rev1=1 rx=0x7f63e0014680 tx=0x7f63f0055f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:05:58.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:05:58.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.567+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:05:58.580 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:05:58.568+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35292 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:05:58.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:05:58.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.569+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:05:58.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.569+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:05:58.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.569+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:05:58.583 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:05:58.583 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:05:58.583 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:58.583 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:05:58.583 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:05:58.584 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:05:58.584 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:05:58.584 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:05:58.584 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:05:58.584 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:05:58.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.574+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:05:58.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.574+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:05:58.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.574+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:05:58.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.574+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:05:58.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:58.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:05:58.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:58.683+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:05:59.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:59.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:05:59.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:05:59.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:00.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:00.241+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:00.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:00.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:00.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:01.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:01.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:01.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:01.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:02.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:02.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:02.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:02.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:03.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:03.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:03.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=12807 cs=0 l=1 rev1=1 rx=0x7f63e0014680 tx=0x7f63f0055f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:03.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 secure :-1 s=READY pgs=12807 cs=0 l=1 rev1=1 rx=0x7f63e0014680 tx=0x7f63f0055f90).stop 2021-03-09T10:06:03.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:03.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:03.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000ca70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:03.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000ca70 unknown :-1 s=CLOSED pgs=12807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:03.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:06:03.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:03.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:03.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:03.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:03.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:03.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:03.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:03.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:06:03.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:03.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023e70 con 0x7f63d8008840 2021-03-09T10:06:03.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:03.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f6400146050 2021-03-09T10:06:03.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.569+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:03.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.570+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:03.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.570+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:03.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.570+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:03.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.570+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:03.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.571+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:03.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.571+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:06:03.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.571+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:03.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.571+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:03.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:03.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:03.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:03.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:03.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:03.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:03.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:03.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:03.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:06:03.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:06:03.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:03.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:03.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0009650 con 0x7f6400146050 2021-03-09T10:06:03.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:03.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:03.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T10:06:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:06:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.572+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.573+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:06:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.573+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12820 cs=0 l=1 rev1=1 rx=0x7f63f0038fa0 tx=0x7f63f0027e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:03.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.573+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:03.590 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:03.575+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35330 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:03.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.575+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:06:03.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.575+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:03.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.575+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:03.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.575+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:03.591 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:03.592 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:03.592 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:03.592 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:03.592 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:03.593 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:03.593 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:03.593 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:03.593 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:03.593 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:03.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.578+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:06:03.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.578+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:03.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.578+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:06:03.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.581+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f6400146050 2021-03-09T10:06:03.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:03.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:03.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:04.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:04.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:04.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:04.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:05.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:05.242+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:05.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:05.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:05.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:05.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:06.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:06.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:06.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:06.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:07.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:07.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:07.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:07.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:08.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12820 cs=0 l=1 rev1=1 rx=0x7f63f0038fa0 tx=0x7f63f0027e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12820 cs=0 l=1 rev1=1 rx=0x7f63f0038fa0 tx=0x7f63f0027e70).stop 2021-03-09T10:06:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:06:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:08.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:08.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:08.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T10:06:08.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:08.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025380 con 0x7f63d8008840 2021-03-09T10:06:08.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:08.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8005e00 2021-03-09T10:06:08.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:08.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:08.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.576+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:08.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.577+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:08.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.577+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:08.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.577+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:08.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.577+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:06:08.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.577+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:08.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.577+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:08.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.578+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:08.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.578+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:08.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.578+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:08.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:08.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:08.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:08.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:08.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:08.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:06:08.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:06:08.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:08.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:08.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028450 con 0x7f63d8005e00 2021-03-09T10:06:08.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:08.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:08.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0009650 con 0x7f63d8005e00 2021-03-09T10:06:08.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:06:08.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:08.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:06:08.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067130 secure :-1 s=READY pgs=12835 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0048a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:08.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.579+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:08.593 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:08.581+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35370 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:08.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.581+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e460 con 0x7f63d8005e00 2021-03-09T10:06:08.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.581+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:08.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.581+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:08.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.581+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:08.595 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:08.595 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:08.595 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:08.595 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:08.596 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:08.596 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:08.596 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:08.596 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:08.597 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:08.597 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:08.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.584+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0088f00 con 0x7f63d8005e00 2021-03-09T10:06:08.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.584+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:08.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.584+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:06:08.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.586+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8005e00 2021-03-09T10:06:08.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:08.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:08.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:08.684+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:09.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:09.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:09.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:10.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:10.243+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:10.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:10.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:10.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:10.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:11.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:11.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:12.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:12.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:12.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:12.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:13.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0067130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:13.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0067130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:13.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067130 secure :-1 s=READY pgs=12835 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0048a20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:13.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067130 secure :-1 s=READY pgs=12835 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0048a20).stop 2021-03-09T10:06:13.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:13.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:13.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:13.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0067130 unknown :-1 s=CLOSED pgs=12835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:13.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:06:13.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:13.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:13.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:13.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:13.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:13.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:13.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:13.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c180 con 0x7f63d8003400 2021-03-09T10:06:13.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:13.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ca70 con 0x7f63d8008840 2021-03-09T10:06:13.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:13.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:06:13.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:13.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:13.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:13.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:13.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:13.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:13.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:06:13.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:13.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.582+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:13.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:13.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:13.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:13.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:13.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:13.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:13.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:13.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:13.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:06:13.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:06:13.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:13.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:13.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079330 con 0x7f6400146050 2021-03-09T10:06:13.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:13.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:13.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028450 con 0x7f6400146050 2021-03-09T10:06:13.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:06:13.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.583+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:13.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.584+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12843 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f006dfb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:13.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.584+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:06:13.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.584+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:13.599 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:13.586+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35408 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:13.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:06:13.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.587+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:13.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.587+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:13.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.587+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:13.600 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:13.601 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:13.601 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:13.601 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:13.601 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:13.601 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:13.602 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:13.602 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:13.602 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:13.602 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:13.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.591+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:06:13.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.591+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:13.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.591+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:06:13.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f6400146050 2021-03-09T10:06:13.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:13.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:13.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:14.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:14.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:14.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:14.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:15.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:15.244+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:15.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:15.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:15.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:15.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:16.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:16.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:16.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:16.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:17.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:17.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:18.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:18.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:18.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12843 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f006dfb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:18.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12843 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f006dfb0).stop 2021-03-09T10:06:18.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:18.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:18.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:18.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:18.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:06:18.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:18.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:18.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:18.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:18.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:18.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:18.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:18.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:06:18.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:18.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f5f0 con 0x7f63d8008840 2021-03-09T10:06:18.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:18.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c080 con 0x7f63d8005e00 2021-03-09T10:06:18.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:18.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:18.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:18.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:18.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.587+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:18.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:18.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:06:18.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:18.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:18.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:18.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:18.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:18.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:18.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:18.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:18.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:18.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:18.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:06:18.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:06:18.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:18.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:18.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:06:18.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:18.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079330 con 0x7f63d8005e00 2021-03-09T10:06:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:06:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.588+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.589+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:06:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.589+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:18.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.589+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021f10 secure :-1 s=READY pgs=12852 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f006d4f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:18.665 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:18.592+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35448 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:18.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e430 con 0x7f63d8005e00 2021-03-09T10:06:18.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.592+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:18.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.592+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:18.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.592+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:18.667 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:18.667 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:18.668 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:18.668 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:18.668 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:18.668 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:18.669 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:18.669 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:18.669 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:18.669 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:18.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8005e00 2021-03-09T10:06:18.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.592+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:18.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.592+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:06:18.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00887f0 con 0x7f63d8005e00 2021-03-09T10:06:18.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:18.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:18.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:18.684+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:19.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:19.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:19.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:19.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:20.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:20.245+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:20.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:20.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:20.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:20.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:21.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:21.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:21.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:21.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:22.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:22.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:23.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021f10 secure :-1 s=READY pgs=12852 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f006d4f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021f10 secure :-1 s=READY pgs=12852 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f006d4f0).stop 2021-03-09T10:06:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:23.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021f10 unknown :-1 s=CLOSED pgs=12852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:06:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:23.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:23.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:23.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:23.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:23.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:23.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c180 con 0x7f63d8003400 2021-03-09T10:06:23.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:23.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028860 con 0x7f63d8008840 2021-03-09T10:06:23.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:23.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017a60 con 0x7f6400146050 2021-03-09T10:06:23.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:23.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:23.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:23.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:23.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.593+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:23.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.594+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:23.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.594+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:06:23.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.594+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:23.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.594+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:23.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.594+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:23.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.594+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:23.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.594+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:23.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:23.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:23.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:23.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:23.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:23.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:06:23.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:06:23.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:23.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:23.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010320 con 0x7f6400146050 2021-03-09T10:06:23.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:23.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:23.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T10:06:23.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:06:23.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.595+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:23.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.596+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12869 cs=0 l=1 rev1=1 rx=0x7f63f0038fd0 tx=0x7f63f0055eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:23.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.596+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:06:23.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.596+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:23.609 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:23.599+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35486 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:23.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002f3b0 con 0x7f6400146050 2021-03-09T10:06:23.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.599+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:23.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.599+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:23.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.599+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:23.611 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:23.611 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:23.612 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:23.613 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:23.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.600+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002f1d0 con 0x7f6400146050 2021-03-09T10:06:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.600+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f006db60 con 0x7f6400146050 2021-03-09T10:06:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.603+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:06:23.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:23.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:23.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:24.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:24.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:24.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:25.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:25.246+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:25.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:25.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:25.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:25.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:26.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:26.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:26.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:26.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:27.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:27.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:27.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:27.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:28.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:28.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:28.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12869 cs=0 l=1 rev1=1 rx=0x7f63f0038fd0 tx=0x7f63f0055eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:28.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12869 cs=0 l=1 rev1=1 rx=0x7f63f0038fd0 tx=0x7f63f0055eb0).stop 2021-03-09T10:06:28.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:28.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:28.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:28.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:28.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:06:28.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0022840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:28.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:28.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:28.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:28.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:28.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:28.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:28.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:06:28.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:28.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8008840 2021-03-09T10:06:28.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:28.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00096a0 con 0x7f63d8005e00 2021-03-09T10:06:28.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.599+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:28.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.600+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:28.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.600+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:28.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:28.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.600+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:28.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0022840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:28.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:06:28.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:28.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:28.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:28.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:28.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:28.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:28.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:28.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:28.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:28.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:28.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:06:28.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:06:28.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:28.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:28.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:06:28.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:28.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:28.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010320 con 0x7f63d8005e00 2021-03-09T10:06:28.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:06:28.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.601+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:28.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.602+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0022840 secure :-1 s=READY pgs=12883 cs=0 l=1 rev1=1 rx=0x7f63e001a9c0 tx=0x7f63f0039f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:28.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.602+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:06:28.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.602+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:28.614 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:28.604+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35526 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:28.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0088e70 con 0x7f63d8005e00 2021-03-09T10:06:28.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.605+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:28.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.605+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:28.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.605+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:28.616 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:28.617 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:28.617 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:28.617 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:28.617 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:28.618 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:28.618 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:28.618 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:28.618 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:28.619 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:28.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0088c70 con 0x7f63d8005e00 2021-03-09T10:06:28.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.605+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:28.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.605+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:06:28.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.607+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f63d8005e00 2021-03-09T10:06:28.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:28.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:28.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:28.687+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:29.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:29.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:29.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:29.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:30.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:30.247+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:30.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:30.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:30.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:30.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:31.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:31.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:31.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:31.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:32.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:32.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:32.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:32.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:33.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0022840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:33.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0022840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:33.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0022840 secure :-1 s=READY pgs=12883 cs=0 l=1 rev1=1 rx=0x7f63e001a9c0 tx=0x7f63f0039f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:33.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0022840 secure :-1 s=READY pgs=12883 cs=0 l=1 rev1=1 rx=0x7f63e001a9c0 tx=0x7f63f0039f90).stop 2021-03-09T10:06:33.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:33.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:33.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0022840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:33.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0022840 unknown :-1 s=CLOSED pgs=12883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:33.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:06:33.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:33.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:33.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:33.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:33.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:33.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:33.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:33.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068e80 con 0x7f63d8003400 2021-03-09T10:06:33.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:33.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014160 con 0x7f63d8008840 2021-03-09T10:06:33.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:33.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f6400146050 2021-03-09T10:06:33.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:33.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:33.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:33.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:33.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:33.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.606+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:33.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.606+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:06:33.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.606+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:33.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.606+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:33.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.607+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:33.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.607+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:33.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.607+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:33.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:33.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:33.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:33.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:33.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:33.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:06:33.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:06:33.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:33.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:33.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002b5b0 con 0x7f6400146050 2021-03-09T10:06:33.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:33.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:33.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00104b0 con 0x7f6400146050 2021-03-09T10:06:33.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:06:33.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:33.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:06:33.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:33.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.608+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12900 cs=0 l=1 rev1=1 rx=0x7f63e001cdb0 tx=0x7f63fc051c90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:33.621 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:33.610+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35564 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:33.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f6400146050 2021-03-09T10:06:33.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.611+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:33.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.611+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:33.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.611+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:33.622 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:33.623 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:33.623 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:33.623 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:33.623 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:33.623 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:33.624 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:33.624 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:33.624 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:33.624 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:33.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.617+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f6400146050 2021-03-09T10:06:33.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.617+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:33.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.617+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f6400146050 2021-03-09T10:06:33.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.617+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:06:33.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:33.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:33.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:34.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:34.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:34.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:34.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:35.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:35.248+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:35.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:35.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:35.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:35.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:36.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:36.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:36.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:36.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:37.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:37.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:37.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:37.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:38.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.610+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:06:38.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.610+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:38.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.610+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12900 cs=0 l=1 rev1=1 rx=0x7f63e001cdb0 tx=0x7f63fc051c90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:38.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12900 cs=0 l=1 rev1=1 rx=0x7f63e001cdb0 tx=0x7f63fc051c90).stop 2021-03-09T10:06:38.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:38.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:38.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:38.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:38.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:06:38.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:38.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:38.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:38.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:38.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:38.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:38.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:38.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:06:38.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:38.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066750 con 0x7f63d8008840 2021-03-09T10:06:38.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:38.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c080 con 0x7f63d8005e00 2021-03-09T10:06:38.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:38.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:38.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:38.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:38.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:38.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:38.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:06:38.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:38.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:38.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:38.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:38.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.611+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:38.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:38.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:38.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:38.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:38.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:38.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:06:38.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:06:38.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:38.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:38.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00119c0 con 0x7f63d8005e00 2021-03-09T10:06:38.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:38.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:38.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002b5b0 con 0x7f63d8005e00 2021-03-09T10:06:38.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:06:38.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.612+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:38.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.613+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:06:38.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.613+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:38.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.613+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00114d0 secure :-1 s=READY pgs=12909 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc03ff80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:38.626 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:38.616+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35604 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:38.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.617+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8005e00 2021-03-09T10:06:38.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.617+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:38.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.617+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:38.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.617+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:38.628 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:38.628 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:38.628 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:38.628 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:38.628 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:38.629 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:38.629 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:38.629 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:38.629 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:38.629 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:38.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.620+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05b170 con 0x7f63d8005e00 2021-03-09T10:06:38.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.620+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:38.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.620+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:06:38.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.622+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8005e00 2021-03-09T10:06:38.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:38.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:38.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:38.688+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:39.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:39.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:39.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:39.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:40.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:40.249+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:40.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:40.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:40.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:40.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:41.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:41.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:41.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:41.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:42.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:42.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:42.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:43.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:43.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:43.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00114d0 secure :-1 s=READY pgs=12909 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc03ff80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:43.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00114d0 secure :-1 s=READY pgs=12909 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc03ff80).stop 2021-03-09T10:06:43.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:43.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:43.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00114d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:43.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00114d0 unknown :-1 s=CLOSED pgs=12909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:43.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:06:43.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:43.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:43.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:43.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:43.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:43.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:43.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:43.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b700 con 0x7f63d8003400 2021-03-09T10:06:43.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:43.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:06:43.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:43.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014120 con 0x7f6400146050 2021-03-09T10:06:43.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.616+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:43.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:43.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:43.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:43.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:43.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:43.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:06:43.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:43.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:43.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:43.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:43.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.617+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:43.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:43.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:43.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:43.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:43.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:43.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:06:43.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:06:43.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:43.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:43.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015c90 con 0x7f6400146050 2021-03-09T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00119c0 con 0x7f6400146050 2021-03-09T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:06:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.618+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:43.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.619+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:06:43.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.619+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:43.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.619+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12923 cs=0 l=1 rev1=1 rx=0x7f63fc06dee0 tx=0x7f63fc01f790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:43.633 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:43.622+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35642 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.623+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c760 con 0x7f6400146050 2021-03-09T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.623+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:43.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.623+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:43.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.623+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:43.635 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:43.635 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:43.635 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:43.636 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:43.637 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:43.638 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:43.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02c760 con 0x7f6400146050 2021-03-09T10:06:43.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.628+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:43.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.628+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:06:43.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f6400146050 2021-03-09T10:06:43.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:43.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:43.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:44.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:44.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:44.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:44.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:45.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:45.250+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:45.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:45.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:45.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:45.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:46.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:46.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:46.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:46.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:47.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:47.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:47.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:47.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:48.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:48.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:48.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12923 cs=0 l=1 rev1=1 rx=0x7f63fc06dee0 tx=0x7f63fc01f790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:48.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12923 cs=0 l=1 rev1=1 rx=0x7f63fc06dee0 tx=0x7f63fc01f790).stop 2021-03-09T10:06:48.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:48.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:48.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:48.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:48.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:06:48.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0011ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:48.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:48.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:48.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:48.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:48.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:48.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:48.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c3c0 con 0x7f63d8003400 2021-03-09T10:06:48.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:48.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014310 con 0x7f63d8008840 2021-03-09T10:06:48.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:48.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:06:48.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.621+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:48.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.622+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:48.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.622+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:48.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.622+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:48.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.622+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:48.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.622+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0011ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:48.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.623+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:06:48.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.623+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:48.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.623+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:48.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.623+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:48.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:48.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:48.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:48.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:48.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:48.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:48.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:48.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:06:48.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:06:48.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:48.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:48.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b090 con 0x7f63d8005e00 2021-03-09T10:06:48.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:48.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:48.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:06:48.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:06:48.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.624+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:48.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.625+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0011ba0 secure :-1 s=READY pgs=12943 cs=0 l=1 rev1=1 rx=0x7f63f001c830 tx=0x7f63f0055dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:48.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.625+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:06:48.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.625+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:48.650 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:48.626+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35682 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:48.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009580 con 0x7f63d8005e00 2021-03-09T10:06:48.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.627+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:48.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.627+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:48.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.628+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:48.653 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:48.653 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:48.654 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:48.654 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:48.655 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:48.655 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:48.655 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:48.656 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:48.656 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:48.657 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:48.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.630+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0038e80 con 0x7f63d8005e00 2021-03-09T10:06:48.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.630+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:48.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.630+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:06:48.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.632+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:06:48.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:48.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:48.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:48.688+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:49.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:49.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:49.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:49.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:50.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:50.250+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:50.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:50.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:50.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:50.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:51.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:51.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:51.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:51.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:52.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:52.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:52.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:53.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0011ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:06:53.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0011ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:53.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0011ba0 secure :-1 s=READY pgs=12943 cs=0 l=1 rev1=1 rx=0x7f63f001c830 tx=0x7f63f0055dc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:53.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0011ba0 secure :-1 s=READY pgs=12943 cs=0 l=1 rev1=1 rx=0x7f63f001c830 tx=0x7f63f0055dc0).stop 2021-03-09T10:06:53.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:53.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:53.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0011ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:53.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0011ba0 unknown :-1 s=CLOSED pgs=12943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:53.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:06:53.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:53.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:53.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:53.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:53.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:53.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:53.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:53.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c080 con 0x7f63d8003400 2021-03-09T10:06:53.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:53.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e300 con 0x7f63d8008840 2021-03-09T10:06:53.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:53.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:06:53.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:53.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:53.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:53.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:53.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.627+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:53.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.628+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:53.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.628+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:06:53.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.628+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:53.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.628+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:53.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.628+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:53.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.628+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:53.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.628+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:53.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:53.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:53.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:53.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:53.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:53.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:06:53.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:06:53.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:53.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:53.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001910 con 0x7f6400146050 2021-03-09T10:06:53.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:53.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:53.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b090 con 0x7f6400146050 2021-03-09T10:06:53.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:06:53.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.629+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:53.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.630+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:06:53.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.630+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:53.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.630+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12956 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0039ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:53.643 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:53.632+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35720 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:53.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.633+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:06:53.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.633+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:53.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.633+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:53.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.633+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:53.645 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:53.645 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:53.645 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:53.645 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:53.645 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:53.646 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:53.646 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:53.646 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:53.646 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:53.647 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:53.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.637+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f6400146050 2021-03-09T10:06:53.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.637+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:53.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.637+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f6400146050 2021-03-09T10:06:53.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.637+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:06:53.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:53.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:53.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:54.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:54.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:54.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:54.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:55.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:55.251+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:06:55.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:55.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:55.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:55.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:56.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:56.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:56.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:56.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:57.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:57.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:57.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:57.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:58.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:06:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:06:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12956 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0039ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:06:58.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12956 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0039ea0).stop 2021-03-09T10:06:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:58.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:06:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:06:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12956 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:06:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:06:58.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:58.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:06:58.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:58.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:06:58.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:06:58.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:06:58.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:58.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014120 con 0x7f63d8003400 2021-03-09T10:06:58.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:58.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007be10 con 0x7f63d8008840 2021-03-09T10:06:58.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:06:58.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ae80 con 0x7f63d8005e00 2021-03-09T10:06:58.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:06:58.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:06:58.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.633+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:58.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.634+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:06:58.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.634+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:06:58.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.635+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:06:58.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.635+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:06:58.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.635+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:06:58.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.635+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:06:58.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:06:58.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:06:58.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:06:58.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:06:58.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:58.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:58.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:06:58.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:06:58.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:06:58.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:06:58.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:58.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:58.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:06:58.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:06:58.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:06:58.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015c90 con 0x7f63d8005e00 2021-03-09T10:06:58.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:06:58.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.636+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:58.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.637+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 secure :-1 s=READY pgs=12970 cs=0 l=1 rev1=1 rx=0x7f63fc052000 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:06:58.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.637+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:06:58.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.637+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:06:58.648 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:06:58.638+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35760 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:06:58.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.638+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:06:58.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.638+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:06:58.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.638+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:06:58.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.638+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:06:58.650 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:06:58.650 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:06:58.650 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:58.651 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:06:58.651 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:06:58.651 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:06:58.651 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:06:58.651 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:06:58.652 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:06:58.652 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:06:58.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.640+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:06:58.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.640+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:06:58.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.640+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:06:58.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.641+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc066d50 con 0x7f63d8005e00 2021-03-09T10:06:58.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:58.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:06:58.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:58.690+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:06:59.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:59.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:06:59.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:06:59.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:00.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:00.252+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:00.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:00.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:00.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:00.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:01.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:01.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:01.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:01.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:02.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:02.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:02.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:02.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:03.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:03.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:03.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 secure :-1 s=READY pgs=12970 cs=0 l=1 rev1=1 rx=0x7f63fc052000 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:03.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 secure :-1 s=READY pgs=12970 cs=0 l=1 rev1=1 rx=0x7f63fc052000 tx=0x7f63fc04a6d0).stop 2021-03-09T10:07:03.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:03.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:03.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:03.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021340 unknown :-1 s=CLOSED pgs=12970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:03.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:07:03.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:03.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:03.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:03.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:03.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:03.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:03.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:03.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:07:03.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:03.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e500 con 0x7f63d8008840 2021-03-09T10:07:03.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:03.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f6400146050 2021-03-09T10:07:03.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:03.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:03.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:03.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:03.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.639+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:03.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.640+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:03.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.640+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:07:03.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.640+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:03.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.640+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:03.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.640+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:03.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.640+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:03.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.640+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:03.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:03.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:03.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:03.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:03.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:03.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:03.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:03.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:03.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:03.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c800 con 0x7f6400146050 2021-03-09T10:07:03.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:03.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:03.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001910 con 0x7f6400146050 2021-03-09T10:07:03.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:03.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.642+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:03.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.643+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12985 cs=0 l=1 rev1=1 rx=0x7f63f007be00 tx=0x7f63f004dab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:03.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.643+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:07:03.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.643+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:03.655 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:03.644+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35798 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:03.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007fea0 con 0x7f6400146050 2021-03-09T10:07:03.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.644+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:03.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.644+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:03.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.644+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:03.656 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:03.657 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:03.657 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:03.657 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:03.657 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:03.657 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:03.658 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:03.658 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:03.658 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:03.658 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:03.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.646+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007fea0 con 0x7f6400146050 2021-03-09T10:07:03.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.646+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:03.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.646+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:07:03.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.647+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T10:07:03.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:03.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:03.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:04.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:04.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:04.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:04.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:05.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:05.253+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:05.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:05.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:05.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:05.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:06.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:06.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:06.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:06.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:07.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:07.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:07.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:07.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:08.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:07:08.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:08.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12985 cs=0 l=1 rev1=1 rx=0x7f63f007be00 tx=0x7f63f004dab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:08.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=12985 cs=0 l=1 rev1=1 rx=0x7f63f007be00 tx=0x7f63f004dab0).stop 2021-03-09T10:07:08.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:08.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:08.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:08.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=12985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:08.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:07:08.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:08.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:08.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:08.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:08.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:08.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:08.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:08.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:07:08.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:08.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ae80 con 0x7f63d8008840 2021-03-09T10:07:08.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:08.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014120 con 0x7f63d8005e00 2021-03-09T10:07:08.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.644+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:08.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:08.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:08.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:08.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:08.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:08.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:07:08.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:08.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:08.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:08.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:08.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.645+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:08.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:08.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:08.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:08.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:08.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:08.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:08.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:08.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:08.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:08.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f63d8005e00 2021-03-09T10:07:08.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:08.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:08.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c800 con 0x7f63d8005e00 2021-03-09T10:07:08.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:08.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:08.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:07:08.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 secure :-1 s=READY pgs=12996 cs=0 l=1 rev1=1 rx=0x7f63f007cfe0 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:08.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.647+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:08.660 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:08.649+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35838 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:08.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0042960 con 0x7f63d8005e00 2021-03-09T10:07:08.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.650+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:08.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.650+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:08.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.650+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:08.661 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:08.662 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:08.662 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:08.662 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:08.662 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:08.663 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:08.663 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:08.663 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:08.663 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:08.663 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:08.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0042960 con 0x7f63d8005e00 2021-03-09T10:07:08.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.652+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:08.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.653+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:07:08.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.654+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:07:08.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:08.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:08.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:08.691+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:09.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:09.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:09.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:09.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:10.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:10.254+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:10.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:10.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:10.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:10.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:11.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:11.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:11.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:11.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:12.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:12.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:12.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:12.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:13.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0008c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:13.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0008c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:13.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 secure :-1 s=READY pgs=12996 cs=0 l=1 rev1=1 rx=0x7f63f007cfe0 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:13.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 secure :-1 s=READY pgs=12996 cs=0 l=1 rev1=1 rx=0x7f63f007cfe0 tx=0x7f63f004d180).stop 2021-03-09T10:07:13.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:13.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:13.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0008c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:13.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 unknown :-1 s=CLOSED pgs=12996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:13.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:07:13.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:13.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:13.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:13.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:13.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:13.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:13.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:13.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197a0 con 0x7f63d8003400 2021-03-09T10:07:13.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:13.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8008840 2021-03-09T10:07:13.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:13.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e500 con 0x7f6400146050 2021-03-09T10:07:13.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:13.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:13.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.650+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:13.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.651+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:13.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.651+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:13.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.651+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:13.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.651+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:07:13.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.651+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:13.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.651+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:13.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.651+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:13.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.651+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:13.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.651+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:13.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:13.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:13.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:13.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:13.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:13.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:13.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:13.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:13.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:13.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00653b0 con 0x7f6400146050 2021-03-09T10:07:13.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:13.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:13.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f6400146050 2021-03-09T10:07:13.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:13.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.652+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:13.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.653+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13010 cs=0 l=1 rev1=1 rx=0x7f63f0011490 tx=0x7f63f0039f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:13.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.653+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:07:13.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.653+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:13.667 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:13.655+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35876 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:13.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.655+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036c40 con 0x7f6400146050 2021-03-09T10:07:13.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.655+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:13.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.655+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:13.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.655+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:13.669 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:13.669 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:13.669 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:13.670 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:13.670 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:13.670 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:13.670 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:13.671 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:13.671 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:13.671 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:13.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0036c40 con 0x7f6400146050 2021-03-09T10:07:13.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.658+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:13.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.658+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:07:13.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.659+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f6400146050 2021-03-09T10:07:13.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:13.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:13.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:14.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:14.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:14.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:14.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:15.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:15.255+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:15.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:15.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:15.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:15.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:16.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:16.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:16.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:16.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:17.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:17.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:17.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:17.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:18.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.655+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:07:18.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.655+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:18.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.655+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13010 cs=0 l=1 rev1=1 rx=0x7f63f0011490 tx=0x7f63f0039f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:18.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13010 cs=0 l=1 rev1=1 rx=0x7f63f0011490 tx=0x7f63f0039f80).stop 2021-03-09T10:07:18.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:18.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:18.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:18.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13010 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:18.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:07:18.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:18.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:18.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:18.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:07:18.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:18.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008a30 con 0x7f63d8008840 2021-03-09T10:07:18.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:18.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8005e00 2021-03-09T10:07:18.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:18.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:18.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.656+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:18.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.657+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:18.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.657+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:18.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:18.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.657+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:07:18.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.657+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:18.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.657+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:18.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.657+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:18.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.657+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:18.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.657+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:18.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:18.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:18.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:18.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:18.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:18.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:18.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:18.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:18.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:18.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b9c0 con 0x7f63d8005e00 2021-03-09T10:07:18.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:18.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:18.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00653b0 con 0x7f63d8005e00 2021-03-09T10:07:18.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:18.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.658+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:18.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.659+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 secure :-1 s=READY pgs=13020 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0013b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:18.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.659+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:07:18.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.659+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:18.673 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:18.661+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35916 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:18.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8005e00 2021-03-09T10:07:18.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.661+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:18.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.661+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:18.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.661+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:18.675 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:18.675 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:18.676 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:18.676 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:18.676 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:18.676 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:18.676 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:18.677 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:18.677 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:18.677 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:18.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f006d2c0 con 0x7f63d8005e00 2021-03-09T10:07:18.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.661+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:18.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.661+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:07:18.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007fb30 con 0x7f63d8005e00 2021-03-09T10:07:18.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:18.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:18.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:18.692+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:19.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:19.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:19.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:19.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:20.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:20.256+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:20.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:20.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:20.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:20.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:21.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:21.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:21.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:21.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:22.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:22.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:22.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:23.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0008c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:23.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0008c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:23.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 secure :-1 s=READY pgs=13020 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0013b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:23.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 secure :-1 s=READY pgs=13020 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0013b90).stop 2021-03-09T10:07:23.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:23.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:23.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0008c20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:23.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008c20 unknown :-1 s=CLOSED pgs=13020 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:23.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:07:23.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:23.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:23.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:23.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:23.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:23.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:23.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:23.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010500 con 0x7f63d8003400 2021-03-09T10:07:23.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:23.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8008840 2021-03-09T10:07:23.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:23.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f6400146050 2021-03-09T10:07:23.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.661+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:23.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.662+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:23.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.662+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:23.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.662+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:23.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.662+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:23.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.662+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:23.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.662+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:07:23.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.662+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:23.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.662+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:23.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:23.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:23.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:23.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016ea0 con 0x7f6400146050 2021-03-09T10:07:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b9c0 con 0x7f6400146050 2021-03-09T10:07:23.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:23.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.664+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:23.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.665+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13039 cs=0 l=1 rev1=1 rx=0x7f63f0056050 tx=0x7f63f007fdb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:23.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.665+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:07:23.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.665+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:23.677 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:23.667+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35954 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:23.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0042df0 con 0x7f6400146050 2021-03-09T10:07:23.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.667+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:23.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.667+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:23.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.667+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:23.679 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:23.680 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:23.681 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.671+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055e90 con 0x7f6400146050 2021-03-09T10:07:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.671+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:23.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.671+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:07:23.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.671+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f6400146050 2021-03-09T10:07:23.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:23.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:23.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:24.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:24.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:24.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:24.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:25.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:25.257+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:25.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:25.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:25.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:25.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:26.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:26.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:26.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:26.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:27.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:27.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:27.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:27.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:28.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:28.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:28.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13039 cs=0 l=1 rev1=1 rx=0x7f63f0056050 tx=0x7f63f007fdb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:28.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13039 cs=0 l=1 rev1=1 rx=0x7f63f0056050 tx=0x7f63f007fdb0).stop 2021-03-09T10:07:28.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:28.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:28.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:28.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:28.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:07:28.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:28.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:28.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:28.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:28.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:28.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:28.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:28.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b340 con 0x7f63d8003400 2021-03-09T10:07:28.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:28.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197a0 con 0x7f63d8008840 2021-03-09T10:07:28.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:28.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:07:28.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.667+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:28.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:28.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:28.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:28.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:28.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:28.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:07:28.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:28.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:28.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:28.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:28.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.668+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:28.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:28.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:28.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:28.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:28.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:28.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:07:28.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:07:28.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:28.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:28.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f63d8005e00 2021-03-09T10:07:28.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:28.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:28.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:07:28.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:07:28.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.669+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:28.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.670+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 secure :-1 s=READY pgs=13047 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc0712f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:28.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.670+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:07:28.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.670+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:28.683 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:28.672+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 35994 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:28.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8005e00 2021-03-09T10:07:28.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.672+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:28.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.672+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:28.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.672+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:28.684 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:28.685 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:28.685 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:28.685 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:28.685 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:28.686 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:28.686 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:28.686 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:28.686 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:28.686 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:28.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8005e00 2021-03-09T10:07:28.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.673+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:28.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.673+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:07:28.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.674+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8005e00 2021-03-09T10:07:28.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:28.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:28.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:28.694+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:29.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:29.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:29.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:29.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:30.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:30.258+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:30.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:30.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:30.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:30.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:31.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:31.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:31.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:31.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:32.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:32.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:32.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:32.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:33.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:33.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:33.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 secure :-1 s=READY pgs=13047 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc0712f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:33.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 secure :-1 s=READY pgs=13047 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc0712f0).stop 2021-03-09T10:07:33.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:33.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:33.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:33.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 unknown :-1 s=CLOSED pgs=13047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:33.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:07:33.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:33.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:33.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:33.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:33.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:33.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:33.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:33.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00258b0 con 0x7f63d8003400 2021-03-09T10:07:33.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:33.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d220 con 0x7f63d8008840 2021-03-09T10:07:33.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:33.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00098a0 con 0x7f6400146050 2021-03-09T10:07:33.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:33.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:33.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:33.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:33.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.673+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:33.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.674+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:33.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.674+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:07:33.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.674+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:33.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.674+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:33.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.674+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:33.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.674+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:33.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.674+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:33.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:33.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:33.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:33.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:33.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:33.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:33.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:33.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:33.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:33.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T10:07:33.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:33.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:33.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016ea0 con 0x7f6400146050 2021-03-09T10:07:33.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:33.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.675+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:33.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.676+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13066 cs=0 l=1 rev1=1 rx=0x7f63f0039f50 tx=0x7f63f002d290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:33.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.676+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:07:33.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.676+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:33.690 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:33.678+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36030 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:33.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f6400146050 2021-03-09T10:07:33.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.679+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:33.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.679+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:33.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.679+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:33.691 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:33.692 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:33.692 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:33.692 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:33.692 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:33.692 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:33.693 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:33.693 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:33.693 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:33.693 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:33.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0042df0 con 0x7f6400146050 2021-03-09T10:07:33.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.687+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:33.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.687+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:07:33.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f6400146050 2021-03-09T10:07:33.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:33.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:33.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:34.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:34.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:34.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:34.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:35.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:35.259+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:35.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:35.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:35.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:35.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:36.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:36.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:36.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:36.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:37.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:37.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:37.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:37.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:38.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:38.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:38.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13066 cs=0 l=1 rev1=1 rx=0x7f63f0039f50 tx=0x7f63f002d290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:38.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13066 cs=0 l=1 rev1=1 rx=0x7f63f0039f50 tx=0x7f63f002d290).stop 2021-03-09T10:07:38.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:38.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:38.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:38.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:38.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:07:38.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:38.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:38.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:38.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:38.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:38.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:38.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:38.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:07:38.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:38.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197a0 con 0x7f63d8008840 2021-03-09T10:07:38.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:38.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cdd0 con 0x7f63d8005e00 2021-03-09T10:07:38.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:38.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:38.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:38.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:38.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:38.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:38.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:07:38.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:38.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.679+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:38.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.680+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:38.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.680+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:38.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.680+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:38.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:38.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:38.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:38.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:38.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:38.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:38.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:38.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:38.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:38.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025300 con 0x7f63d8005e00 2021-03-09T10:07:38.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:38.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:38.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:07:38.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:38.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:38.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:07:38.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 secure :-1 s=READY pgs=13080 cs=0 l=1 rev1=1 rx=0x7f63f007e5f0 tx=0x7f63f0030290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:38.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.681+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:38.695 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:38.684+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36072 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:38.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e660 con 0x7f63d8005e00 2021-03-09T10:07:38.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.685+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:38.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.685+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:38.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.685+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:38.697 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:38.697 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:38.697 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:38.697 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:38.698 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:38.698 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:38.698 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:38.699 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:38.699 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:38.699 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:38.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.689+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e430 con 0x7f63d8005e00 2021-03-09T10:07:38.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.689+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:38.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00314b0 con 0x7f63d8005e00 2021-03-09T10:07:38.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.690+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:07:38.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:38.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:38.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:38.695+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:39.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:39.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:39.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:39.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:40.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:40.260+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:40.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:40.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:40.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:40.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:41.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:41.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:41.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:41.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:42.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:42.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:42.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:42.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:43.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:43.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:43.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 secure :-1 s=READY pgs=13080 cs=0 l=1 rev1=1 rx=0x7f63f007e5f0 tx=0x7f63f0030290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:43.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 secure :-1 s=READY pgs=13080 cs=0 l=1 rev1=1 rx=0x7f63f007e5f0 tx=0x7f63f0030290).stop 2021-03-09T10:07:43.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:43.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:43.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:43.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 unknown :-1 s=CLOSED pgs=13080 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:43.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:07:43.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:43.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:43.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:43.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:43.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:43.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:43.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:43.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8003400 2021-03-09T10:07:43.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:43.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b8c0 con 0x7f63d8008840 2021-03-09T10:07:43.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:43.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069080 con 0x7f6400146050 2021-03-09T10:07:43.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:43.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:43.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:43.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:43.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.684+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:43.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.685+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:43.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.685+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:07:43.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.685+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:43.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.685+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:43.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.685+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:43.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.685+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:43.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.685+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:43.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:43.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:43.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:43.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:43.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:43.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:43.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:43.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:43.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f6400146050 2021-03-09T10:07:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:43.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025300 con 0x7f6400146050 2021-03-09T10:07:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.686+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.687+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13091 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007a430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:43.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.687+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f6400146050 2021-03-09T10:07:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.687+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:43.701 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:43.689+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36110 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f006d4d0 con 0x7f6400146050 2021-03-09T10:07:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.690+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:43.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.690+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:43.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.690+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:43.703 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:43.703 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:43.703 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:43.704 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:43.704 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:43.704 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:43.704 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:43.704 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:43.705 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.693+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f006d4d0 con 0x7f6400146050 2021-03-09T10:07:43.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.693+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.693+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f6400146050 2021-03-09T10:07:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.693+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:07:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:43.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:43.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:44.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:44.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:44.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:44.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:45.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:45.261+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:45.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:45.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:45.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:45.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:46.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:46.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:46.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:46.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:47.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:47.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:47.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:47.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:48.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:07:48.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:48.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13091 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007a430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:48.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 secure :-1 s=READY pgs=13091 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007a430).stop 2021-03-09T10:07:48.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:48.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:48.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:48.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:48.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:07:48.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:48.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:48.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:48.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:48.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:48.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:48.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:48.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:07:48.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:48.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197a0 con 0x7f63d8008840 2021-03-09T10:07:48.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:48.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cb40 con 0x7f63d8005e00 2021-03-09T10:07:48.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:48.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:48.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:07:48.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:48.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:48.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:48.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:48.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:48.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.690+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:48.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:48.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:48.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:48.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:48.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:48.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:07:48.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:07:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015c30 con 0x7f63d8005e00 2021-03-09T10:07:48.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:48.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:48.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f63d8005e00 2021-03-09T10:07:48.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:07:48.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.691+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:48.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.692+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 secure :-1 s=READY pgs=13101 cs=0 l=1 rev1=1 rx=0x7f63e0019b70 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:48.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.692+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:07:48.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.692+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:48.705 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:48.695+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36149 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:48.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8005e00 2021-03-09T10:07:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.696+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.696+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.696+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:48.707 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:48.708 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:48.708 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:48.708 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:48.708 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:48.708 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:48.708 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:48.709 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:07:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.696+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.696+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:07:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.697+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:07:48.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:48.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8005e00 2021-03-09T10:07:49.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:49.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:49.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:49.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:50.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:50.262+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:50.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:50.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:50.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:50.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:51.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:51.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:51.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:51.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:52.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:52.698+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:07:52.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:52.698+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:53.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:53.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:53.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 secure :-1 s=READY pgs=13101 cs=0 l=1 rev1=1 rx=0x7f63e0019b70 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:53.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 secure :-1 s=READY pgs=13101 cs=0 l=1 rev1=1 rx=0x7f63e0019b70 tx=0x7f63fc009990).stop 2021-03-09T10:07:53.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:53.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:53.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:53.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b280 unknown :-1 s=CLOSED pgs=13101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:53.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:07:53.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:53.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8008840 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:53.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:53.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:53.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:53.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f6400146050 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:53.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:53.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069080 con 0x7f63d8003400 2021-03-09T10:07:53.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:53.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b8c0 con 0x7f63d8008840 2021-03-09T10:07:53.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:53.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cdd0 con 0x7f6400146050 2021-03-09T10:07:53.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:53.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:53.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:53.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:53.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.696+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:53.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:53.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f6400146050 auth_method 0 2021-03-09T10:07:53.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:53.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:53.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient(hunting): tick 2021-03-09T10:07:53.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 monclient(hunting): continuing hunt 2021-03-09T10:07:53.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient(hunting): _reopen_session rank -1 2021-03-09T10:07:53.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2021-03-09T10:07:53.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f6400146050 0x7f63e0064240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:53.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 msgr2=0x7f63e000dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:53.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8008840 0x7f63e000dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:53.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0027570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:53.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0027570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:53.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:07:53.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63d8008c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:53.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:53.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63d8008290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:53.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:53.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63d8009530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:53.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:53.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:53.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069080 con 0x7f63d8003400 2021-03-09T10:07:53.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:53.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b8c0 con 0x7f63d8009130 2021-03-09T10:07:53.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:53.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cdd0 con 0x7f63d8003840 2021-03-09T10:07:53.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f63ecff9700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:53.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63d8009530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:53.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63d8009530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:53.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.698+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63d8008290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:53.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.698+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63d8008290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:53.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.698+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63d8008c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:53.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.698+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:07:53.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.698+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:53.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.698+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:53.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.698+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:53.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.698+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:53.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.698+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:53.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:53.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63d8008290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:53.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63d8008290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:53.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63d8009530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:53.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63d8009530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:53.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:53.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:53.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1.5 2021-03-09T10:07:53.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:53.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002efc0 con 0x7f63d8003840 2021-03-09T10:07:53.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:53.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:53.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:07:53.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:53.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.699+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:53.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.700+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:07:53.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.700+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:53.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.700+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63d8008c40 secure :-1 s=READY pgs=13120 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:53.721 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:53.701+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36188 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:53.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:07:53.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.702+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:53.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.702+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:53.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.702+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:53.723 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:53.723 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:53.723 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:53.724 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:53.724 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:53.724 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:53.724 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:53.724 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:53.725 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:53.725 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:53.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:07:53.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.706+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:53.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.706+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:07:53.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:53.707+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:07:55.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:55.263+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:07:58.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63d8008c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:07:58.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63d8008c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:07:58.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63d8008c40 secure :-1 s=READY pgs=13120 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:07:58.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63d8008c40 secure :-1 s=READY pgs=13120 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f0000c30).stop 2021-03-09T10:07:58.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:58.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:07:58.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63d8008c40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:07:58.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63d8008c40 unknown :-1 s=CLOSED pgs=13120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:07:58.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:07:58.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:58.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:07:58.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:58.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:07:58.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:07:58.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:07:58.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:58.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:07:58.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:58.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197a0 con 0x7f63d8009130 2021-03-09T10:07:58.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:07:58.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:07:58.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:07:58.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:07:58.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:58.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.702+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:07:58.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.703+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:07:58.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.704+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:07:58.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.704+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:07:58.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.704+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:07:58.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.704+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:07:58.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.704+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:07:58.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.704+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:07:58.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.704+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:07:58.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:07:58.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:58.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:58.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:07:58.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:07:58.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:07:58.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:07:58.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1.5 2021-03-09T10:07:58.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:58.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028d80 con 0x7f63d8005e00 2021-03-09T10:07:58.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:07:58.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:07:58.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002efc0 con 0x7f63d8005e00 2021-03-09T10:07:58.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:07:58.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:07:58.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b210 secure :-1 s=READY pgs=13134 cs=0 l=1 rev1=1 rx=0x7f63f0030260 tx=0x7f63f0036d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:07:58.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:07:58.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.705+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:07:58.718 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:07:58.707+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36226 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:07:58.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040440 con 0x7f63d8005e00 2021-03-09T10:07:58.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.708+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:07:58.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.708+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:07:58.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.708+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:07:58.719 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:07:58.719 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:07:58.720 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:58.720 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:07:58.720 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:07:58.720 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:07:58.721 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:07:58.721 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:07:58.721 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:07:58.721 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:07:58.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.710+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:07:58.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.710+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:07:58.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.710+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:07:58.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:07:58.710+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8005e00 2021-03-09T10:08:00.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:00.264+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:00.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:00.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:00.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:00.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:00.448+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:01.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:01.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:01.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:02.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:02.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:02.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:02.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:03.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:03.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:03.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:03.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:03.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b210 secure :-1 s=READY pgs=13134 cs=0 l=1 rev1=1 rx=0x7f63f0030260 tx=0x7f63f0036d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:03.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b210 secure :-1 s=READY pgs=13134 cs=0 l=1 rev1=1 rx=0x7f63f0030260 tx=0x7f63f0036d20).stop 2021-03-09T10:08:03.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:03.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:03.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:03.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b210 unknown :-1 s=CLOSED pgs=13134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:03.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:08:03.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:03.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:03.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:03.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:03.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:03.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:03.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:03.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cb40 con 0x7f63d8003400 2021-03-09T10:08:03.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:03.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:08:03.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:03.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a50 con 0x7f63d8003840 2021-03-09T10:08:03.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:03.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:03.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:03.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b700 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:03.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.708+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:08:03.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:03.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:03.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:03.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:03.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:03.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:03.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:03.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:03.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:03.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:03.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:03.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:03.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:03.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:03.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:08:03.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:03.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:03.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.709+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028d80 con 0x7f63d8003840 2021-03-09T10:08:03.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.710+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:03.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.710+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:03.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.710+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:08:03.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.710+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:03.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.710+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 secure :-1 s=READY pgs=13144 cs=0 l=1 rev1=1 rx=0x7f63f007bea0 tx=0x7f63f007a250).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:03.735 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:03.713+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36266 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:03.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:08:03.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.714+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:03.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.714+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:03.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.714+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:03.737 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:03.737 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:03.738 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:03.738 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:03.738 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:03.738 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:03.739 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:03.739 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:03.739 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:03.739 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:03.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.717+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:08:03.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.717+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:03.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.717+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:08:03.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:03.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0073340 con 0x7f63d8003840 2021-03-09T10:08:04.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:04.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:04.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:04.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:05.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:05.265+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:05.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:05.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:05.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:05.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:06.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:06.448+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:06.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:06.448+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:07.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:07.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:07.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:07.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:08.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:08.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:08.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:08.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:08.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 secure :-1 s=READY pgs=13144 cs=0 l=1 rev1=1 rx=0x7f63f007bea0 tx=0x7f63f007a250).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:08.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 secure :-1 s=READY pgs=13144 cs=0 l=1 rev1=1 rx=0x7f63f007bea0 tx=0x7f63f007a250).stop 2021-03-09T10:08:08.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:08.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:08.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007af80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:08.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 unknown :-1 s=CLOSED pgs=13144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:08.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:08:08.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:08.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:08.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:08.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:08.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:08.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:08.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:08.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cdd0 con 0x7f63d8003400 2021-03-09T10:08:08.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:08.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b8c0 con 0x7f63d8009130 2021-03-09T10:08:08.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:08.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:08:08.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:08.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:08.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:08.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:08.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:08.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.715+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:08.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.715+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:08:08.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.715+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:08.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.715+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:08.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.715+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:08.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.716+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:08.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.716+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:08.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:08.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:08.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:08.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:08.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:08.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:08.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:08.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:08.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:08.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002efc0 con 0x7f63d8005e00 2021-03-09T10:08:08.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:08.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:08.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:08:08.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:08.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.717+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:08.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.718+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007cbb0 secure :-1 s=READY pgs=13162 cs=0 l=1 rev1=1 rx=0x7f63f0056050 tx=0x7f63f0031f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:08.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.718+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:08:08.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.718+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:08.729 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:08.720+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36304 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:08.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0039e40 con 0x7f63d8005e00 2021-03-09T10:08:08.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.721+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:08.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.721+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:08.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.721+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:08.731 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:08.731 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:08.732 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:08.732 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:08.732 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:08.733 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:08.733 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:08.733 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:08.733 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:08.733 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:08.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.723+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0039e40 con 0x7f63d8005e00 2021-03-09T10:08:08.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.723+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:08.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.723+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:08:08.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:08.724+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007fb30 con 0x7f63d8005e00 2021-03-09T10:08:09.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:09.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:09.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:09.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:10.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:10.266+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:10.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:10.449+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:10.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:10.449+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:10.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:10.449+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:11.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:11.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:11.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:11.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:12.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:12.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:12.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:12.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:13.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:13.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:13.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:13.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:13.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007cbb0 secure :-1 s=READY pgs=13162 cs=0 l=1 rev1=1 rx=0x7f63f0056050 tx=0x7f63f0031f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:13.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007cbb0 secure :-1 s=READY pgs=13162 cs=0 l=1 rev1=1 rx=0x7f63f0056050 tx=0x7f63f0031f80).stop 2021-03-09T10:08:13.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:13.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:13.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007cbb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:13.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007cbb0 unknown :-1 s=CLOSED pgs=13162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:13.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:08:13.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:13.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:13.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:13.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:13.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:13.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:13.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:13.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:08:13.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:13.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:08:13.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:13.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a490 con 0x7f63d8003840 2021-03-09T10:08:13.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:13.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:13.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:13.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:13.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:13.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:13.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:08:13.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:13.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.721+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:13.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.722+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:13.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.722+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:13.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.722+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:13.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:13.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:13.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:13.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:13.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:13.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:13.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:13.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:13.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:13.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8003840 2021-03-09T10:08:13.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:13.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:13.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002efc0 con 0x7f63d8003840 2021-03-09T10:08:13.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:13.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.723+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:13.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.724+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 secure :-1 s=READY pgs=13176 cs=0 l=1 rev1=1 rx=0x7f63f0067040 tx=0x7f63f0012400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:13.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.724+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:08:13.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.724+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:13.737 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:13.727+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36344 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:13.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0032540 con 0x7f63d8003840 2021-03-09T10:08:13.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.727+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:13.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.727+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:13.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.727+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:13.739 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:13.739 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:13.740 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:13.740 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:13.740 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:13.740 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:13.740 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:13.741 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:13.741 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:13.741 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:13.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0032360 con 0x7f63d8003840 2021-03-09T10:08:13.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.728+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:13.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.728+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:08:13.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:13.729+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0073bc0 con 0x7f63d8003840 2021-03-09T10:08:14.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:14.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:14.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:14.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:15.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:15.267+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:15.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:15.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:15.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:15.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:16.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:16.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:16.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:16.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:17.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:17.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:17.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:17.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:18.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:18.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:18.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:18.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:18.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 secure :-1 s=READY pgs=13176 cs=0 l=1 rev1=1 rx=0x7f63f0067040 tx=0x7f63f0012400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:18.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 secure :-1 s=READY pgs=13176 cs=0 l=1 rev1=1 rx=0x7f63f0067040 tx=0x7f63f0012400).stop 2021-03-09T10:08:18.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:18.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:18.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007af80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:18.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007af80 unknown :-1 s=CLOSED pgs=13176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:18.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:08:18.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:18.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:18.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:18.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:18.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:18.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:18.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:18.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:08:18.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:08:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b8c0 con 0x7f63d8005e00 2021-03-09T10:08:18.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:18.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:18.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:18.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:18.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:18.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:08:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.727+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:18.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:18.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:18.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:18.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:18.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:18.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:18.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:18.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:18.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:18.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:08:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T10:08:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.728+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:18.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.729+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13185 cs=0 l=1 rev1=1 rx=0x7f63f00576d0 tx=0x7f63f007a330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:18.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.729+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:08:18.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.729+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:18.746 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:18.732+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36382 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:18.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.732+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:08:18.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.732+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:18.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.732+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:18.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.732+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:18.748 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:18.749 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:18.749 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:18.749 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:18.749 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:18.749 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:18.750 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:18.750 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:18.750 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:18.750 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:18.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.732+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8005e00 2021-03-09T10:08:18.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.732+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:18.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.732+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:08:18.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:18.734+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007fde0 con 0x7f63d8005e00 2021-03-09T10:08:19.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:19.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:19.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:19.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:20.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:20.268+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:20.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:20.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:20.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:20.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:20.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:20.451+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:21.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:21.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:21.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:21.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:22.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:22.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:22.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:22.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:23.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.450+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:23.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.450+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:23.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:23.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:23.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.732+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13185 cs=0 l=1 rev1=1 rx=0x7f63f00576d0 tx=0x7f63f007a330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:23.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.732+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13185 cs=0 l=1 rev1=1 rx=0x7f63f00576d0 tx=0x7f63f007a330).stop 2021-03-09T10:08:23.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:08:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:23.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:23.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:23.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:23.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:23.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:23.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:23.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a50 con 0x7f63d8003400 2021-03-09T10:08:23.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:23.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:08:23.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:23.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003840 2021-03-09T10:08:23.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:23.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:23.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:23.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:23.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:23.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.733+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:23.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:08:23.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:23.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:23.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:23.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:23.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:23.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e4b0 con 0x7f63d8003840 2021-03-09T10:08:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:08:23.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:23.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.734+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:23.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.735+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:08:23.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007cbb0 secure :-1 s=READY pgs=13201 cs=0 l=1 rev1=1 rx=0x7f63f00557a0 tx=0x7f63f0013ba0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:23.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.735+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:23.749 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:23.738+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36420 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:23.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:08:23.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.738+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:23.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.738+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:23.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.738+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:23.750 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:23.751 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:23.751 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:23.751 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:23.751 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:23.751 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:23.752 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:23.752 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:23.752 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:23.752 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:23.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004c870 con 0x7f63d8003840 2021-03-09T10:08:23.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.739+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:23.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.739+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:08:23.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:23.740+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007fd40 con 0x7f63d8003840 2021-03-09T10:08:24.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:24.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:24.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:24.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:25.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:25.269+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:25.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:25.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:25.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:25.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:26.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:26.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:26.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:26.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:27.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:27.451+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:27.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:27.451+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:28.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:28.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:28.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:08:28.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:28.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007cbb0 secure :-1 s=READY pgs=13201 cs=0 l=1 rev1=1 rx=0x7f63f00557a0 tx=0x7f63f0013ba0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:28.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007cbb0 secure :-1 s=READY pgs=13201 cs=0 l=1 rev1=1 rx=0x7f63f00557a0 tx=0x7f63f0013ba0).stop 2021-03-09T10:08:28.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:28.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:28.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007cbb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:28.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007cbb0 unknown :-1 s=CLOSED pgs=13201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:28.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:08:28.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:28.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:28.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:28.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:28.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:28.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:28.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:28.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266f0 con 0x7f63d8003400 2021-03-09T10:08:28.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:28.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024860 con 0x7f63d8009130 2021-03-09T10:08:28.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:28.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b8c0 con 0x7f63d8005e00 2021-03-09T10:08:28.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:28.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:28.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:28.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:28.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.739+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:28.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:28.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:08:28.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:28.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:28.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:28.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:28.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:28.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:28.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:28.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:28.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:28.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:28.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:28.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:28.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:28.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:28.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028d80 con 0x7f63d8005e00 2021-03-09T10:08:28.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:28.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:28.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e4b0 con 0x7f63d8005e00 2021-03-09T10:08:28.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:28.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.740+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:28.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.741+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:08:28.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.741+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:28.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.741+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13214 cs=0 l=1 rev1=1 rx=0x7f63f007a300 tx=0x7f63f004dab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:28.753 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:28.744+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36458 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:28.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.744+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027430 con 0x7f63d8005e00 2021-03-09T10:08:28.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.744+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:28.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.744+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:28.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.744+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:28.755 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:28.755 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:28.756 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:28.756 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:28.756 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:28.756 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:28.757 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:28.757 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:28.757 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:28.757 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:28.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.744+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8005e00 2021-03-09T10:08:28.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.744+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:28.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.745+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:08:28.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:28.746+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007fd40 con 0x7f63d8005e00 2021-03-09T10:08:29.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:29.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:29.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:29.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:30.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:30.270+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:30.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:30.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:30.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:30.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:30.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:30.452+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:31.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:31.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:31.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:31.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:32.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:32.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:32.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:32.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:33.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:33.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:33.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:08:33.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:33.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13214 cs=0 l=1 rev1=1 rx=0x7f63f007a300 tx=0x7f63f004dab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:33.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13214 cs=0 l=1 rev1=1 rx=0x7f63f007a300 tx=0x7f63f004dab0).stop 2021-03-09T10:08:33.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:33.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:33.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:33.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:33.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:08:33.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:33.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:33.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:33.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:33.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:33.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:33.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:33.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:08:33.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:33.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:08:33.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:33.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009470 con 0x7f63d8003840 2021-03-09T10:08:33.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:33.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:33.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:33.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:33.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.745+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:33.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.746+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:33.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.746+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:08:33.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.746+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:33.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.746+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:33.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.746+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:33.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.746+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:33.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.746+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:33.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.747+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:33.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.747+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:33.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.747+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:33.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.747+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:33.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.747+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:33.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.747+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:33.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.747+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:33.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:33.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:33.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:08:33.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:33.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:33.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028d80 con 0x7f63d8003840 2021-03-09T10:08:33.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:33.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:33.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 secure :-1 s=READY pgs=13231 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f007eef0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:33.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:08:33.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.748+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:33.762 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:33.750+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36500 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:33.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002f1d0 con 0x7f63d8003840 2021-03-09T10:08:33.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.751+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:33.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.751+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:33.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.751+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:33.764 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:33.764 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:33.764 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:33.765 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:33.765 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:33.765 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:33.765 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:33.766 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:33.766 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:33.766 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:33.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.754+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:08:33.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.754+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:33.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.754+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:08:33.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:33.754+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0057390 con 0x7f63d8003840 2021-03-09T10:08:34.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:34.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:34.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:34.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:35.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:35.271+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:35.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:35.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:35.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:35.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:36.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:36.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:36.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:36.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:37.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:37.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:37.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:37.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:38.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.452+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:38.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.452+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:38.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:38.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:38.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 secure :-1 s=READY pgs=13231 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f007eef0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:38.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 secure :-1 s=READY pgs=13231 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f007eef0).stop 2021-03-09T10:08:38.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:38.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:38.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:38.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 unknown :-1 s=CLOSED pgs=13231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:38.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:08:38.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:38.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:38.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:38.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:38.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:38.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:38.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:38.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024860 con 0x7f63d8003400 2021-03-09T10:08:38.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:38.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e700 con 0x7f63d8009130 2021-03-09T10:08:38.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:38.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b8c0 con 0x7f63d8005e00 2021-03-09T10:08:38.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:38.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:38.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:38.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:38.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.751+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:38.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.752+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:38.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.752+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:08:38.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.752+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:38.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.752+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:38.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:38.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:38.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:38.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:38.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:38.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:38.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:38.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:38.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:38.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:38.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:38.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:38.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d7e0 con 0x7f63d8005e00 2021-03-09T10:08:38.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:38.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:38.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.753+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:08:38.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.754+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:38.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.754+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:38.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.754+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:08:38.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.754+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13247 cs=0 l=1 rev1=1 rx=0x7f63f002f390 tx=0x7f63f0039820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:38.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.754+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:38.768 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:38.757+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36538 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:38.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8005e00 2021-03-09T10:08:38.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.758+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:38.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.758+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:38.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.758+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:38.769 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:38.770 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:38.770 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:38.770 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:38.770 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:38.771 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:38.771 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:38.771 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:38.771 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:38.771 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:38.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.759+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8005e00 2021-03-09T10:08:38.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.759+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:38.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.759+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:08:38.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:38.760+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8005e00 2021-03-09T10:08:39.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:39.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:39.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:39.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:40.272+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:40.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:40.453+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:40.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:40.453+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:40.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:40.453+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:41.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:41.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:41.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:41.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:42.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:42.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:42.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:42.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:43.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:43.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:43.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:08:43.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:43.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13247 cs=0 l=1 rev1=1 rx=0x7f63f002f390 tx=0x7f63f0039820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:43.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13247 cs=0 l=1 rev1=1 rx=0x7f63f002f390 tx=0x7f63f0039820).stop 2021-03-09T10:08:43.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:43.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:43.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:43.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:43.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:08:43.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:43.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:43.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:43.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:43.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:43.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:43.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:43.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a50 con 0x7f63d8003400 2021-03-09T10:08:43.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:43.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:08:43.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:43.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a450 con 0x7f63d8003840 2021-03-09T10:08:43.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:43.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b590 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:43.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:43.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:43.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.758+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:43.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:43.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.759+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:08:43.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.759+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:43.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.759+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:43.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:43.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:43.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:43.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:43.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:43.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:43.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:43.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:43.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:43.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:43.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:43.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:43.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012890 con 0x7f63d8003840 2021-03-09T10:08:43.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:43.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:43.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d7e0 con 0x7f63d8003840 2021-03-09T10:08:43.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:43.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.760+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:43.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.761+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 secure :-1 s=READY pgs=13261 cs=0 l=1 rev1=1 rx=0x7f63f0028830 tx=0x7f63f0006ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:43.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.761+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:08:43.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.761+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:43.776 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:43.763+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36578 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:43.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:08:43.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.764+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:43.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.764+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:43.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.764+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:43.778 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:43.778 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:43.778 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:43.778 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:43.779 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:43.779 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:43.779 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:43.779 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:43.779 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:43.780 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:43.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.765+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:08:43.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.765+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:43.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.765+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:08:43.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:43.767+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63d8003840 2021-03-09T10:08:44.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:44.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:44.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:44.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:45.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:45.273+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:45.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:45.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:45.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:45.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:46.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:46.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:47.454+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:47.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:47.454+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:48.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:48.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:48.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:48.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:48.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 secure :-1 s=READY pgs=13261 cs=0 l=1 rev1=1 rx=0x7f63f0028830 tx=0x7f63f0006ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:48.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 secure :-1 s=READY pgs=13261 cs=0 l=1 rev1=1 rx=0x7f63f0028830 tx=0x7f63f0006ca0).stop 2021-03-09T10:08:48.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:48.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:48.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:48.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079e90 unknown :-1 s=CLOSED pgs=13261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:48.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:08:48.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:48.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:48.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:48.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:48.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:48.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:48.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:48.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b8c0 con 0x7f63d8003400 2021-03-09T10:08:48.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:48.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f040 con 0x7f63d8009130 2021-03-09T10:08:48.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:48.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:08:48.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:48.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:48.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:48.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.765+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:48.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.765+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:48.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.765+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:48.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.765+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:08:48.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.765+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:48.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.765+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:48.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.765+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:48.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.765+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:48.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.765+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:48.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:48.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:48.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:48.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:48.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:48.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:48.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:48.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:48.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:48.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002b990 con 0x7f63d8005e00 2021-03-09T10:08:48.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:48.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:48.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012890 con 0x7f63d8005e00 2021-03-09T10:08:48.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:48.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.767+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:48.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.768+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13274 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f0029370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:48.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.768+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:08:48.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.768+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:48.781 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:48.769+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36616 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:48.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00884c0 con 0x7f63d8005e00 2021-03-09T10:08:48.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.770+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:48.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.770+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:48.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.770+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:48.784 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:48.784 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:48.784 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:48.785 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:48.785 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:48.785 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:48.785 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:48.785 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:48.786 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:48.786 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:48.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T10:08:48.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.771+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:48.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.771+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:08:48.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:48.772+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8005e00 2021-03-09T10:08:49.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:49.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:49.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:49.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:50.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:50.274+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:50.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:50.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:50.455+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:51.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:51.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:52.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:52.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:52.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:52.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:53.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:53.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:53.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:53.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:53.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13274 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f0029370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:53.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13274 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f0029370).stop 2021-03-09T10:08:53.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:53.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:53.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:53.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:53.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:08:53.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:53.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:53.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:53.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:53.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:53.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:53.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:53.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:08:53.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:53.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00018d0 con 0x7f63d8009130 2021-03-09T10:08:53.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:53.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a450 con 0x7f63d8003840 2021-03-09T10:08:53.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:53.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:53.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:08:53.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:53.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:53.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:53.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:53.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:53.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:53.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:53.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:53.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.770+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:53.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:53.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:53.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:53.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:53.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:53.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:08:53.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:08:53.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:53.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:53.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:08:53.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:53.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:53.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015c30 con 0x7f63d8003840 2021-03-09T10:08:53.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:08:53.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.771+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:53.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.772+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:08:53.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.772+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:53.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.772+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=13283 cs=0 l=1 rev1=1 rx=0x7f63e00227a0 tx=0x7f63fc03ff10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:53.785 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:53.775+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36656 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:53.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:08:53.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.775+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:53.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.775+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:53.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.775+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:53.787 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:53.787 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:53.787 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:53.787 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:53.787 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:53.788 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:53.788 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:53.788 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:53.788 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:53.788 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:53.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8003840 2021-03-09T10:08:53.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.775+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:53.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.775+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:08:53.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:53.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8003840 2021-03-09T10:08:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:54.455+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:54.455+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:55.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:55.275+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:08:55.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:55.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:55.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:55.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:56.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:56.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:56.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:56.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:57.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:57.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:57.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:57.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:58.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:58.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:58.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:08:58.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:08:58.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=13283 cs=0 l=1 rev1=1 rx=0x7f63e00227a0 tx=0x7f63fc03ff10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:08:58.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=13283 cs=0 l=1 rev1=1 rx=0x7f63e00227a0 tx=0x7f63fc03ff10).stop 2021-03-09T10:08:58.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:58.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:08:58.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:08:58.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=13283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:08:58.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:08:58.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:58.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:08:58.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:58.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:08:58.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:08:58.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:08:58.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:58.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8003400 2021-03-09T10:08:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:08:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:08:58.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f040 con 0x7f63d8005e00 2021-03-09T10:08:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:08:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:08:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.776+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.777+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:08:58.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.777+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:08:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.777+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:08:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.777+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:08:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.777+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:08:58.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.777+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:08:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.778+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:08:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.778+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:08:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.778+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:08:58.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:08:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:08:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:08:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:08:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:08:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:08:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026670 con 0x7f63d8005e00 2021-03-09T10:08:58.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:08:58.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:08:58.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002b990 con 0x7f63d8005e00 2021-03-09T10:08:58.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:08:58.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.779+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:08:58.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.780+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13303 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:08:58.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.780+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:08:58.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.780+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:08:58.792 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:08:58.781+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36694 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:08:58.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8005e00 2021-03-09T10:08:58.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.782+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:08:58.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.782+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:08:58.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.782+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:08:58.794 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:08:58.794 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:08:58.794 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:58.794 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:08:58.795 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:08:58.795 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:08:58.795 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:08:58.795 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:08:58.796 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:08:58.796 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:08:58.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00481f0 con 0x7f63d8005e00 2021-03-09T10:08:58.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.786+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:08:58.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f63d8005e00 2021-03-09T10:08:58.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:58.787+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:08:59.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:59.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:08:59.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:08:59.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:00.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:00.276+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:00.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:00.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:00.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:00.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:00.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:00.456+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:01.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:01.456+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:01.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:01.456+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:02.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:02.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:02.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:02.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:03.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:03.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:03.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:03.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:03.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13303 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:03.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13303 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f004dd90).stop 2021-03-09T10:09:03.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:03.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:03.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:03.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:03.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:09:03.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:03.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:03.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00156d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:03.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:03.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:03.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:03.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:03.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b690 con 0x7f63d8003400 2021-03-09T10:09:03.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:03.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a50 con 0x7f63d8009130 2021-03-09T10:09:03.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:03.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:09:03.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:03.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:03.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:03.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:03.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.782+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:03.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00156d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:03.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:09:03.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:03.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:03.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:03.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:03.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:03.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:03.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:03.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:03.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:03.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:03.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:09:03.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:09:03.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:03.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:03.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001afe0 con 0x7f63d8003840 2021-03-09T10:09:03.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:03.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:03.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.783+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:09:03.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.784+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:09:03.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.784+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:03.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.784+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00156d0 secure :-1 s=READY pgs=13309 cs=0 l=1 rev1=1 rx=0x7f63e0026830 tx=0x7f63fc066470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:03.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.784+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:09:03.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.785+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:03.798 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:03.787+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36734 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:03.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.788+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f63d8003840 2021-03-09T10:09:03.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.788+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:03.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.788+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:03.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.788+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:03.801 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:03.801 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:03.801 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:03.801 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:03.801 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:03.802 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:03.802 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:03.802 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:03.802 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:03.803 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:03.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.788+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0668d0 con 0x7f63d8003840 2021-03-09T10:09:03.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.788+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:03.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.788+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:09:03.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:03.789+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8003840 2021-03-09T10:09:04.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:04.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:04.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:04.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:05.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:05.277+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:05.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:05.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:05.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:05.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:06.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:06.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:06.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:06.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:07.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:07.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:07.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:07.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:08.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:08.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:08.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00156d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00156d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00156d0 secure :-1 s=READY pgs=13309 cs=0 l=1 rev1=1 rx=0x7f63e0026830 tx=0x7f63fc066470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00156d0 secure :-1 s=READY pgs=13309 cs=0 l=1 rev1=1 rx=0x7f63e0026830 tx=0x7f63fc066470).stop 2021-03-09T10:09:08.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:08.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:08.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00156d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:08.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00156d0 unknown :-1 s=CLOSED pgs=13309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:08.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:09:08.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:08.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:08.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:08.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:08.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00199a0 con 0x7f63d8003400 2021-03-09T10:09:08.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:08.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ee40 con 0x7f63d8009130 2021-03-09T10:09:08.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:08.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bb40 con 0x7f63d8005e00 2021-03-09T10:09:08.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:08.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:08.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:08.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:08.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:08.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:08.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:09:08.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:08.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:08.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:08.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:08.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.788+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:08.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:08.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:08.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:08.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:08.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:08.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:09:08.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:09:08.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:08.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:08.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:09:08.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:08.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:08.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026670 con 0x7f63d8005e00 2021-03-09T10:09:08.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:09:08.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.789+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:08.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.790+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13324 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:08.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.790+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:09:08.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.790+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:08.803 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:08.794+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36772 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:08.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.794+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T10:09:08.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.794+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:08.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.794+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:08.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.794+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:08.804 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:08.805 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:08.805 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:08.805 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:08.805 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:08.805 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:08.805 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:08.806 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:08.806 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:08.806 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:08.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T10:09:08.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.795+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:08.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.795+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:09:08.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:08.796+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007fd40 con 0x7f63d8005e00 2021-03-09T10:09:09.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:09.457+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:09.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:09.457+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:10.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:10.278+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:10.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:10.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:10.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:10.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:10.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:10.458+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:11.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:11.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:11.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:11.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:12.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:12.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:12.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:12.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:13.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:13.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:13.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:13.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:13.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13324 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:13.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13324 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0000bc0).stop 2021-03-09T10:09:13.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:13.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:13.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:13.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:13.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:09:13.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:13.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:13.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:13.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:13.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:13.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:13.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:13.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f040 con 0x7f63d8003400 2021-03-09T10:09:13.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:13.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00128b0 con 0x7f63d8009130 2021-03-09T10:09:13.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:13.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f040 con 0x7f63d8003840 2021-03-09T10:09:13.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:13.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:13.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:13.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:13.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:13.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:13.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:09:13.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:13.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.795+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:13.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.796+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:13.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.796+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:13.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.796+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:13.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:13.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:13.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:13.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:13.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:13.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:09:13.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:09:13.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:13.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:13.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b50 con 0x7f63d8003840 2021-03-09T10:09:13.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:13.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:13.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001afe0 con 0x7f63d8003840 2021-03-09T10:09:13.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:09:13.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:13.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=13340 cs=0 l=1 rev1=1 rx=0x7f63e0079580 tx=0x7f63fc051c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:13.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:09:13.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.797+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:13.809 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:13.800+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36812 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:13.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.800+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc051e90 con 0x7f63d8003840 2021-03-09T10:09:13.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.800+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:13.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.800+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:13.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.800+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:13.811 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:13.811 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:13.811 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:13.811 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:13.811 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:13.812 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:13.812 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:13.812 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:13.812 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:13.813 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:13.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051e90 con 0x7f63d8003840 2021-03-09T10:09:13.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.805+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:13.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.805+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:09:13.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:13.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:09:14.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:14.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:14.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:14.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:15.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:15.279+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:15.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:15.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:15.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:15.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:16.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:16.458+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:16.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:16.458+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:17.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:17.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:17.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:17.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:18.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:18.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:18.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:18.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:18.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=13340 cs=0 l=1 rev1=1 rx=0x7f63e0079580 tx=0x7f63fc051c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:18.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=13340 cs=0 l=1 rev1=1 rx=0x7f63e0079580 tx=0x7f63fc051c60).stop 2021-03-09T10:09:18.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:18.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:18.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:18.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=CLOSED pgs=13340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:18.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:09:18.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:18.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:18.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:18.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:18.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:18.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:18.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:18.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:09:18.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:18.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b690 con 0x7f63d8009130 2021-03-09T10:09:18.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:18.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc60 con 0x7f63d8005e00 2021-03-09T10:09:18.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:18.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:18.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.801+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:18.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.802+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:18.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.802+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:18.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.802+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:18.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.802+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:09:18.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.802+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:18.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.802+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:18.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.802+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:18.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.802+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:18.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.802+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:18.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:18.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:18.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:18.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:18.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:18.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:09:18.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:09:18.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:18.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:18.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011850 con 0x7f63d8005e00 2021-03-09T10:09:18.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:18.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:18.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:09:18.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:09:18.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:18.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:09:18.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:18.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.803+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13351 cs=0 l=1 rev1=1 rx=0x7f63e0043e20 tx=0x7f63f0030260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:18.815 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:18.807+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36850 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:18.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027e90 con 0x7f63d8005e00 2021-03-09T10:09:18.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.807+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:18.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.807+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:18.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.807+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:18.817 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:18.817 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:18.817 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:18.817 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:18.818 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:18.818 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:18.818 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:18.818 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:18.819 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:18.819 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:18.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.811+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027e90 con 0x7f63d8005e00 2021-03-09T10:09:18.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.811+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:18.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.811+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:09:18.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:18.811+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:09:19.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:19.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:19.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:19.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:20.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:20.280+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:20.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:20.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:20.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:20.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:20.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:20.459+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:21.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:21.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:21.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:21.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:22.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:22.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:22.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:22.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.459+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:23.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.459+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:23.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:23.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:23.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13351 cs=0 l=1 rev1=1 rx=0x7f63e0043e20 tx=0x7f63f0030260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:23.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13351 cs=0 l=1 rev1=1 rx=0x7f63e0043e20 tx=0x7f63f0030260).stop 2021-03-09T10:09:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:23.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:23.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:23.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:09:23.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:23.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:23.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:23.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:23.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:23.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:23.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:23.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:09:23.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:23.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f63d8009130 2021-03-09T10:09:23.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:23.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ca50 con 0x7f63d8003840 2021-03-09T10:09:23.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:23.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:23.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:23.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:23.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.807+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:23.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.808+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:23.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.808+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:09:23.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.808+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:23.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.808+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:23.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.808+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:23.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.808+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:23.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.808+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:23.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:23.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:23.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:23.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:23.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:23.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:09:23.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:09:23.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:23.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:23.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:09:23.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:23.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:23.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011850 con 0x7f63d8003840 2021-03-09T10:09:23.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:09:23.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.809+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:23.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.810+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a520 secure :-1 s=READY pgs=13369 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f003ecc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:23.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.810+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:09:23.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.810+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:23.824 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:23.813+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36890 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:23.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:09:23.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.814+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:23.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.814+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:23.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.814+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:23.825 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:23.826 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:23.826 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:23.826 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:23.826 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:23.827 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:23.827 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:23.827 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:23.827 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:23.827 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:23.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:09:23.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.816+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:23.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.816+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:09:23.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:23.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8003840 2021-03-09T10:09:24.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:24.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:24.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:24.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:25.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:25.281+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:25.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:25.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:25.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:25.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:26.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:26.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:26.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:26.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:27.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:27.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:27.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:27.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:28.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:28.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:28.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:28.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:28.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a520 secure :-1 s=READY pgs=13369 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f003ecc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:28.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a520 secure :-1 s=READY pgs=13369 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f003ecc0).stop 2021-03-09T10:09:28.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:28.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:28.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:28.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a520 unknown :-1 s=CLOSED pgs=13369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:28.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:09:28.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:28.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:28.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:28.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:28.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:28.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:28.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:28.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f040 con 0x7f63d8003400 2021-03-09T10:09:28.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:28.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00199a0 con 0x7f63d8009130 2021-03-09T10:09:28.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.814+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:28.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:09:28.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.814+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:28.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:28.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.814+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:28.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:28.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.814+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:28.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:28.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.815+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:09:28.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.815+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:28.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.815+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:28.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.815+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:28.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.815+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:28.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.815+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:28.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:28.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:28.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:28.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:09:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:09:28.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d840 con 0x7f63d8005e00 2021-03-09T10:09:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:28.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:09:28.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:09:28.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.816+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:28.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.817+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13384 cs=0 l=1 rev1=1 rx=0x7f63f003ec90 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:28.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.817+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:09:28.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.817+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:28.829 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:28.818+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36928 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027430 con 0x7f63d8005e00 2021-03-09T10:09:28.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.819+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.819+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.819+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:28.832 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:28.832 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:28.832 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:28.832 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:28.833 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:28.833 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:28.833 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:28.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8005e00 2021-03-09T10:09:28.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.819+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:28.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.819+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:09:28.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:28.820+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8005e00 2021-03-09T10:09:29.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:29.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:29.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:29.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:30.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:30.282+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:30.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:30.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:30.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:30.460+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:30.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:30.460+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:31.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:31.460+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:31.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:31.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:32.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:32.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:32.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:32.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:33.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:33.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:33.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:33.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:33.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13384 cs=0 l=1 rev1=1 rx=0x7f63f003ec90 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:33.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13384 cs=0 l=1 rev1=1 rx=0x7f63f003ec90 tx=0x7f63f0000c30).stop 2021-03-09T10:09:33.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:33.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:33.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:33.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:33.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:09:33.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:33.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:33.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:33.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:33.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:33.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:33.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:33.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:09:33.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:33.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63d8009130 2021-03-09T10:09:33.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:33.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017720 con 0x7f63d8003840 2021-03-09T10:09:33.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:33.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:33.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.819+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:33.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:33.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:33.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:33.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:09:33.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:33.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:33.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:33.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:33.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:33.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:33.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.820+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:33.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:33.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:33.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:33.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:09:33.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:09:33.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:33.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:33.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018980 con 0x7f63d8003840 2021-03-09T10:09:33.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:33.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:33.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d840 con 0x7f63d8003840 2021-03-09T10:09:33.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:09:33.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:33.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c160 secure :-1 s=READY pgs=13392 cs=0 l=1 rev1=1 rx=0x7f63f007b650 tx=0x7f63f003ee10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:33.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:09:33.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.821+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:33.838 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:33.825+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 36968 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:33.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b490 con 0x7f63d8003840 2021-03-09T10:09:33.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.825+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:33.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.825+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:33.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.825+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:33.840 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:33.840 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:33.840 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:33.840 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:33.840 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:33.841 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:33.841 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:33.841 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:33.841 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:33.841 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:33.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:09:33.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.825+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:33.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.825+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:09:33.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:33.826+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8003840 2021-03-09T10:09:34.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:34.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:34.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:34.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:35.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:35.283+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:35.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:35.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:35.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:35.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:36.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:36.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:36.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:36.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:37.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:37.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:37.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:37.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:38.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:38.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:38.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:38.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:38.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c160 secure :-1 s=READY pgs=13392 cs=0 l=1 rev1=1 rx=0x7f63f007b650 tx=0x7f63f003ee10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:38.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c160 secure :-1 s=READY pgs=13392 cs=0 l=1 rev1=1 rx=0x7f63f007b650 tx=0x7f63f003ee10).stop 2021-03-09T10:09:38.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:38.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:38.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:38.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c160 unknown :-1 s=CLOSED pgs=13392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:38.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:09:38.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:38.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:38.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:38.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:38.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:38.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:38.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:38.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:09:38.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:38.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f040 con 0x7f63d8009130 2021-03-09T10:09:38.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:38.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00199a0 con 0x7f63d8005e00 2021-03-09T10:09:38.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.825+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:38.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.826+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:38.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.826+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:38.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.826+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:38.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.826+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:38.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.827+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:38.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.827+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:09:38.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.827+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:38.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.827+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:38.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.827+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:38.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.827+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:38.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.827+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:38.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:38.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:38.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:38.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:38.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:38.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:09:38.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:09:38.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:38.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:38.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:09:38.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:38.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:38.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b50 con 0x7f63d8005e00 2021-03-09T10:09:38.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:09:38.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.828+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:38.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.829+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13414 cs=0 l=1 rev1=1 rx=0x7f63e0048e80 tx=0x7f63fc03ff10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:38.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.829+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:09:38.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.829+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:38.839 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:38.831+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37006 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:38.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8005e00 2021-03-09T10:09:38.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.832+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:38.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.832+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:38.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.832+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:38.841 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:38.841 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:38.841 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:38.842 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:38.842 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:38.842 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:38.842 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:38.843 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:38.843 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:38.843 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:38.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.835+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8005e00 2021-03-09T10:09:38.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.835+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:38.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.835+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:09:38.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:38.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:09:39.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:39.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:39.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:39.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:40.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:40.284+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:40.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:40.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:40.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:40.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:40.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:40.462+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:41.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:41.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:41.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:41.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:42.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:42.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:42.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:42.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:43.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:43.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:43.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:09:43.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:43.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13414 cs=0 l=1 rev1=1 rx=0x7f63e0048e80 tx=0x7f63fc03ff10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:43.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13414 cs=0 l=1 rev1=1 rx=0x7f63e0048e80 tx=0x7f63fc03ff10).stop 2021-03-09T10:09:43.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:43.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:43.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:43.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:43.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:09:43.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:43.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:43.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:43.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:43.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:43.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:43.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:43.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a300 con 0x7f63d8003400 2021-03-09T10:09:43.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:43.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b380 con 0x7f63d8009130 2021-03-09T10:09:43.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:43.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:09:43.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:43.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:43.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.831+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:43.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.832+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:43.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.832+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:43.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.833+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:43.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.833+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:09:43.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.833+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:43.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.833+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:43.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:43.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:43.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:43.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:43.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:43.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:43.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:43.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:43.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:09:43.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:09:43.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:43.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:43.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001c850 con 0x7f63d8003840 2021-03-09T10:09:43.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:43.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:43.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018980 con 0x7f63d8003840 2021-03-09T10:09:43.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:09:43.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.835+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:43.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067c50 secure :-1 s=READY pgs=13430 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f00302d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:43.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.836+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:09:43.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.836+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:43.847 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:43.836+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37046 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:43.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:09:43.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.836+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:43.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.836+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:43.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.836+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:43.848 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:43.849 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:43.849 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:43.849 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:43.849 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:43.849 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:43.850 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:43.850 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:43.850 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:43.850 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:43.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.837+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8003840 2021-03-09T10:09:43.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.837+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:43.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.837+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:09:43.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:43.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f63d8003840 2021-03-09T10:09:44.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:44.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:44.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:44.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:45.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:45.284+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:45.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:45.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:45.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:45.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:46.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:46.461+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:46.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:46.461+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:47.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:47.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:47.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:47.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:48.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:48.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:48.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.836+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:48.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.836+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:48.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067c50 secure :-1 s=READY pgs=13430 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f00302d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:48.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067c50 secure :-1 s=READY pgs=13430 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f00302d0).stop 2021-03-09T10:09:48.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:48.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:48.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:48.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067c50 unknown :-1 s=CLOSED pgs=13430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:48.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:09:48.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:48.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:48.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:48.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:48.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:48.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:48.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:48.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028210 con 0x7f63d8003400 2021-03-09T10:09:48.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:48.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065770 con 0x7f63d8009130 2021-03-09T10:09:48.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:48.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00199a0 con 0x7f63d8005e00 2021-03-09T10:09:48.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:48.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:48.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:48.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:48.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:09:48.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:48.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:48.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:48.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.837+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:48.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:48.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:48.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:48.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:48.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:48.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:48.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:48.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:48.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:09:48.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:09:48.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:48.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:48.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e160 con 0x7f63d8005e00 2021-03-09T10:09:48.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:48.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:48.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:09:48.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:09:48.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:48.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:09:48.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13433 cs=0 l=1 rev1=1 rx=0x7f63fc02bc10 tx=0x7f63fc009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:48.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.838+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:48.860 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:48.842+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37084 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:48.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02ba50 con 0x7f63d8005e00 2021-03-09T10:09:48.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.842+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:48.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.842+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:48.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.842+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:48.862 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:48.862 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:48.862 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:48.862 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:48.863 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:48.863 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:48.863 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:48.863 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:48.863 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:48.864 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:48.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02b820 con 0x7f63d8005e00 2021-03-09T10:09:48.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.842+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:48.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.842+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:09:48.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:48.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:09:49.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:49.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:49.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:49.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:50.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:50.285+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:50.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:50.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:50.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:50.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:50.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:50.462+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:51.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:51.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:51.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:51.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:52.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:52.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:52.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:52.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:53.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.462+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:53.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.462+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:53.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:53.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:53.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13433 cs=0 l=1 rev1=1 rx=0x7f63fc02bc10 tx=0x7f63fc009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:53.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13433 cs=0 l=1 rev1=1 rx=0x7f63fc02bc10 tx=0x7f63fc009a70).stop 2021-03-09T10:09:53.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:53.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:09:53.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:53.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:09:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:53.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f040 con 0x7f63d8009130 2021-03-09T10:09:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017720 con 0x7f63d8003840 2021-03-09T10:09:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:53.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:53.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.843+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:53.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.843+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:53.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:53.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.843+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:09:53.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.843+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:53.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.843+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:53.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:53.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:53.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:53.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:53.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:53.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:53.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:53.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:53.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:09:53.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:09:53.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:53.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:53.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001afc0 con 0x7f63d8003840 2021-03-09T10:09:53.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:53.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:53.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001c850 con 0x7f63d8003840 2021-03-09T10:09:53.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:09:53.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:53.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ac80 secure :-1 s=READY pgs=13453 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00066f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:53.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:09:53.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.845+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:53.858 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:53.847+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37124 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:53.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:09:53.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.848+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:53.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.848+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:53.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.848+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:53.860 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:53.860 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:53.861 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:53.861 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:53.861 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:53.861 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:53.861 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:53.862 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:53.862 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:53.862 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:53.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:09:53.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.852+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:53.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.853+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:09:53.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:53.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ab60 con 0x7f63d8003840 2021-03-09T10:09:54.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:54.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:54.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:54.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:55.286+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:09:55.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:55.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:55.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:55.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:56.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:56.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:56.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:57.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:57.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:57.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:58.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:58.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:58.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:09:58.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:09:58.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ac80 secure :-1 s=READY pgs=13453 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00066f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:09:58.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ac80 secure :-1 s=READY pgs=13453 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00066f0).stop 2021-03-09T10:09:58.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:09:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006ac80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:09:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ac80 unknown :-1 s=CLOSED pgs=13453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:09:58.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:09:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:09:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:58.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:09:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:09:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:09:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:58.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:09:58.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:58.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ca40 con 0x7f63d8009130 2021-03-09T10:09:58.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:09:58.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63d8005e00 2021-03-09T10:09:58.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:09:58.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:09:58.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:58.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:09:58.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:09:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.849+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:09:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.849+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:09:58.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.849+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:09:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.849+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:09:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.849+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:09:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.849+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:09:58.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.849+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:09:58.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:09:58.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:58.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:58.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:09:58.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:09:58.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:09:58.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:09:58.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:09:58.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:58.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b830 con 0x7f63d8005e00 2021-03-09T10:09:58.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:09:58.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:09:58.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e160 con 0x7f63d8005e00 2021-03-09T10:09:58.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:09:58.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.850+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:09:58.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13467 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc05df80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:09:58.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.851+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:09:58.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.851+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:09:58.865 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:09:58.853+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37160 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:09:58.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023a20 con 0x7f63d8005e00 2021-03-09T10:09:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.854+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:09:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.854+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:09:58.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.854+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:09:58.868 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:09:58.868 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:09:58.868 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:58.868 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:09:58.868 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:09:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:09:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:09:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:09:58.869 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:09:58.870 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:09:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023e80 con 0x7f63d8005e00 2021-03-09T10:09:58.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.855+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:09:58.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.855+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:09:58.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:58.856+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc051d80 con 0x7f63d8005e00 2021-03-09T10:09:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:59.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:09:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:09:59.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:00.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:00.287+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:00.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:00.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:00.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:00.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:00.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:00.463+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:01.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:01.463+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:01.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:01.463+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:02.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:02.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:02.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:02.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:03.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:03.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:03.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:03.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:03.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13467 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc05df80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:03.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13467 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc05df80).stop 2021-03-09T10:10:03.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:03.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:03.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:03.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:03.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:10:03.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:03.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:03.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:03.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:03.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:03.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:03.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:03.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8003400 2021-03-09T10:10:03.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:03.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63d8009130 2021-03-09T10:10:03.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:03.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:10:03.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:03.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:03.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:03.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.854+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:03.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:03.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:03.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:10:03.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:03.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:03.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:03.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:03.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:03.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:03.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:03.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:03.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:03.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:03.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:10:03.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:10:03.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:03.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:03.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:10:03.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:03.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:03.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001afc0 con 0x7f63d8003840 2021-03-09T10:10:03.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:10:03.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:03.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:10:03.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=13476 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f007e430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:03.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.856+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:03.871 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:03.859+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37202 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:03.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004cc30 con 0x7f63d8003840 2021-03-09T10:10:03.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.860+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:03.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.860+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:03.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.860+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:03.873 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:03.873 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:03.874 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:03.874 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:03.874 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:03.874 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:03.875 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:03.875 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:03.875 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:03.875 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:03.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:10:03.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.861+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:03.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.861+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:10:03.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:03.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:10:04.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:04.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:04.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:04.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:05.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:05.288+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:05.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:05.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:05.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:05.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:06.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:06.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:06.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:06.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:07.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:07.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:07.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:07.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:08.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.464+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:08.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.464+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:08.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:08.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:08.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=13476 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f007e430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:08.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=13476 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f007e430).stop 2021-03-09T10:10:08.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:08.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:08.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:08.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=CLOSED pgs=13476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:08.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:10:08.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:08.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:08.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:08.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:08.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:08.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:08.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:08.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00199a0 con 0x7f63d8003400 2021-03-09T10:10:08.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:08.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:10:08.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:08.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00247a0 con 0x7f63d8005e00 2021-03-09T10:10:08.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:08.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:08.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:08.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:08.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:08.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.861+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:08.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.861+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:10:08.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.861+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:08.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.861+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:08.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.862+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:08.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.862+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:08.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.862+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:08.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:08.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:08.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:08.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:08.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:08.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:10:08.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:10:08.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:08.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:08.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0009490 con 0x7f63d8005e00 2021-03-09T10:10:08.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:08.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:08.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b830 con 0x7f63d8005e00 2021-03-09T10:10:08.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:10:08.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:08.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13499 cs=0 l=1 rev1=1 rx=0x7f63e0043440 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:08.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:10:08.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.863+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:08.875 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:08.865+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37240 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:08.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058830 con 0x7f63d8005e00 2021-03-09T10:10:08.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.866+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:08.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.866+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:08.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.866+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:08.877 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:08.877 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:08.878 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:08.878 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:08.878 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:08.878 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:08.879 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:08.879 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:08.879 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:08.879 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:08.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058830 con 0x7f63d8005e00 2021-03-09T10:10:08.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.868+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:08.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.868+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:10:08.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:08.869+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:10:09.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:09.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:09.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:09.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:10.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:10.289+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:10.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:10.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:10.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:10.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:10.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:10.465+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:11.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:11.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:11.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:11.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:12.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:12.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:13.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:13.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:13.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:13.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:13.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13499 cs=0 l=1 rev1=1 rx=0x7f63e0043440 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:13.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13499 cs=0 l=1 rev1=1 rx=0x7f63e0043440 tx=0x7f63fc003a50).stop 2021-03-09T10:10:13.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:13.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:13.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:13.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:13.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:10:13.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:13.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:13.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:13.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:13.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:13.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:13.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:13.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8003400 2021-03-09T10:10:13.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:13.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8009130 2021-03-09T10:10:13.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:13.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012da0 con 0x7f63d8003840 2021-03-09T10:10:13.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:13.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:13.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:13.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:13.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:10:13.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:13.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:13.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:13.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.866+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:13.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:13.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:13.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:13.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:13.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:13.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:13.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:13.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:13.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:10:13.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:10:13.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:13.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:13.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c4d0 con 0x7f63d8003840 2021-03-09T10:10:13.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:13.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:13.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:10:13.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:10:13.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.867+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:13.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.868+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:10:13.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.868+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:13.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.868+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a4a0 secure :-1 s=READY pgs=13505 cs=0 l=1 rev1=1 rx=0x7f63e000c090 tx=0x7f63f004d110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:13.884 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:13.871+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37280 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:13.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009570 con 0x7f63d8003840 2021-03-09T10:10:13.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.872+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:13.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.872+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:13.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.872+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:13.886 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:13.886 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:13.886 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:13.887 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:13.887 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:13.887 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:13.887 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:13.888 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:13.888 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:13.888 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:13.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009570 con 0x7f63d8003840 2021-03-09T10:10:13.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.874+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:13.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.874+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:10:13.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:13.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007bb00 con 0x7f63d8003840 2021-03-09T10:10:14.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:14.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:14.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:14.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:15.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:15.290+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:15.465+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:15.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:15.465+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:16.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:16.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:16.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:16.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:17.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:17.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:17.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:17.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:18.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:18.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:18.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a4a0 secure :-1 s=READY pgs=13505 cs=0 l=1 rev1=1 rx=0x7f63e000c090 tx=0x7f63f004d110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:18.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a4a0 secure :-1 s=READY pgs=13505 cs=0 l=1 rev1=1 rx=0x7f63e000c090 tx=0x7f63f004d110).stop 2021-03-09T10:10:18.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:18.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:18.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:18.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a4a0 unknown :-1 s=CLOSED pgs=13505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:18.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:10:18.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:18.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:18.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:18.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:18.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:18.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:18.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:18.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022bf0 con 0x7f63d8003400 2021-03-09T10:10:18.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:18.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:10:18.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:18.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079210 con 0x7f63d8005e00 2021-03-09T10:10:18.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.871+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:18.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:18.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:18.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:18.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:18.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:18.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:10:18.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:18.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:18.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:18.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:18.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.872+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:18.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:18.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:18.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:18.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:18.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:18.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:10:18.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:10:18.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:18.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:18.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:10:18.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:18.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:18.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c4d0 con 0x7f63d8005e00 2021-03-09T10:10:18.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:10:18.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.873+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:18.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13516 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0041da0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:18.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.874+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:10:18.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.874+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:18.887 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:18.877+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37318 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:18.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63d8005e00 2021-03-09T10:10:18.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.878+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:18.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.878+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:18.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.878+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:18.888 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:18.889 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:18.889 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:18.889 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:18.889 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:18.889 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:18.890 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:18.890 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:18.890 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:18.890 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:18.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8005e00 2021-03-09T10:10:18.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.882+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:18.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8005e00 2021-03-09T10:10:18.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:18.883+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:10:19.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:19.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:19.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:19.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:20.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:20.291+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:20.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:20.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:20.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:20.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:20.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:20.466+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:21.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:21.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:21.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:21.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:22.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:22.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:22.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:22.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:23.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.466+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:23.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.466+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:23.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:23.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13516 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0041da0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13516 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0041da0).stop 2021-03-09T10:10:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:23.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:10:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:23.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:23.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197e0 con 0x7f63d8003400 2021-03-09T10:10:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a750 con 0x7f63d8009130 2021-03-09T10:10:23.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:10:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.877+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:23.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:10:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:23.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:23.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:23.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:10:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:10:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:23.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bf40 con 0x7f63d8003840 2021-03-09T10:10:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:10:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:10:23.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.878+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023ce0 secure :-1 s=READY pgs=13531 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f003e260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.879+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:10:23.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.879+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:23.893 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:23.883+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37358 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002f3e0 con 0x7f63d8003840 2021-03-09T10:10:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.883+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.883+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:23.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.883+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:23.895 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:23.896 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:23.896 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:23.896 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:23.896 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:23.896 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8003840 2021-03-09T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.883+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.883+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:10:23.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:23.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f002f1d0 con 0x7f63d8003840 2021-03-09T10:10:24.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:24.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:24.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:24.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:25.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:25.292+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:25.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:25.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:25.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:25.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:26.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:26.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:26.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:26.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:27.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:27.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:27.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:27.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:28.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:28.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:28.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:28.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023ce0 secure :-1 s=READY pgs=13531 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f003e260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:28.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023ce0 secure :-1 s=READY pgs=13531 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f003e260).stop 2021-03-09T10:10:28.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:28.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:28.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023ce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:28.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023ce0 unknown :-1 s=CLOSED pgs=13531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:28.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:10:28.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:28.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:28.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:28.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:28.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:28.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:28.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:28.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197e0 con 0x7f63d8003400 2021-03-09T10:10:28.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:28.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00018d0 con 0x7f63d8009130 2021-03-09T10:10:28.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:28.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012ab0 con 0x7f63d8005e00 2021-03-09T10:10:28.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:28.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:28.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.883+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:28.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:28.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:28.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:28.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:10:28.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:28.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:28.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:28.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:28.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:28.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:28.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:28.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:28.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:28.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:28.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:10:28.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:10:28.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:28.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:28.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014320 con 0x7f63d8005e00 2021-03-09T10:10:28.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:28.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:28.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bf40 con 0x7f63d8005e00 2021-03-09T10:10:28.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:10:28.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.884+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:28.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13544 cs=0 l=1 rev1=1 rx=0x7f63f0036510 tx=0x7f63f004dab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:28.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.885+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:10:28.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.885+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:28.898 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:28.888+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37396 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:28.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036350 con 0x7f63d8005e00 2021-03-09T10:10:28.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.888+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:28.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.888+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:28.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.888+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:28.900 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:28.900 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:28.900 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:28.901 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:28.901 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:28.901 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:28.901 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:28.901 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:28.902 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:28.902 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:28.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8005e00 2021-03-09T10:10:28.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.889+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:28.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.889+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:10:28.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:28.890+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f63d8005e00 2021-03-09T10:10:29.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:29.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:29.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:29.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:30.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:30.293+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:30.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:30.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:30.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:30.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:30.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:30.467+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:31.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:31.467+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:31.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:31.467+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:32.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:32.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:32.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:32.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:33.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:33.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:33.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:10:33.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:33.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13544 cs=0 l=1 rev1=1 rx=0x7f63f0036510 tx=0x7f63f004dab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:33.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13544 cs=0 l=1 rev1=1 rx=0x7f63f0036510 tx=0x7f63f004dab0).stop 2021-03-09T10:10:33.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:33.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:33.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:33.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:33.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:10:33.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:33.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:33.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:33.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:33.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:33.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:33.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:33.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:10:33.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:33.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197e0 con 0x7f63d8009130 2021-03-09T10:10:33.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:33.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022980 con 0x7f63d8003840 2021-03-09T10:10:33.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:33.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:33.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:33.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:33.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.889+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:33.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.890+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:33.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.890+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:10:33.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.890+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:33.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.890+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:33.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:33.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:33.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:33.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:33.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:33.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:33.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:33.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:33.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:10:33.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:10:33.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:33.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:33.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:10:33.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.891+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:33.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.892+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:33.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.892+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014320 con 0x7f63d8003840 2021-03-09T10:10:33.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.892+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:10:33.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.892+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:33.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.892+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007c3e0 secure :-1 s=READY pgs=13563 cs=0 l=1 rev1=1 rx=0x7f63f004c600 tx=0x7f63f00395b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:33.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.892+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:10:33.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.892+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:33.904 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:33.894+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37436 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:33.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004c670 con 0x7f63d8003840 2021-03-09T10:10:33.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.897+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:33.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.897+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:33.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.897+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:33.906 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:33.906 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:33.907 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:33.907 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:33.907 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:33.907 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:33.907 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:33.908 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:33.908 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:33.908 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:33.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004c440 con 0x7f63d8003840 2021-03-09T10:10:33.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.898+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:33.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.898+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:10:33.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:33.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f63d8003840 2021-03-09T10:10:34.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:34.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:34.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:34.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:35.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:35.294+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:35.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:35.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:35.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:35.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:36.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:36.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:36.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:37.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:37.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:37.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:37.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:38.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:38.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:38.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:38.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:38.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007c3e0 secure :-1 s=READY pgs=13563 cs=0 l=1 rev1=1 rx=0x7f63f004c600 tx=0x7f63f00395b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:38.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007c3e0 secure :-1 s=READY pgs=13563 cs=0 l=1 rev1=1 rx=0x7f63f004c600 tx=0x7f63f00395b0).stop 2021-03-09T10:10:38.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:38.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:38.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007c3e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:38.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007c3e0 unknown :-1 s=CLOSED pgs=13563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:38.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:10:38.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:38.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:38.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:38.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:38.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:38.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:38.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:38.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012ab0 con 0x7f63d8003400 2021-03-09T10:10:38.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:38.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197e0 con 0x7f63d8009130 2021-03-09T10:10:38.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:38.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00657b0 con 0x7f63d8005e00 2021-03-09T10:10:38.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:38.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:38.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.895+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:38.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.896+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:38.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.896+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:38.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.896+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:38.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.896+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:10:38.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.896+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:38.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.896+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:38.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:38.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.897+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:38.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.897+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:38.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:38.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:38.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:38.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:38.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:38.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:10:38.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:10:38.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:38.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:38.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b70 con 0x7f63d8005e00 2021-03-09T10:10:38.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:38.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:38.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0009490 con 0x7f63d8005e00 2021-03-09T10:10:38.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:10:38.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:38.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13579 cs=0 l=1 rev1=1 rx=0x7f63fc02bff0 tx=0x7f63fc052fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:38.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:10:38.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.898+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:38.910 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:38.900+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37474 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:38.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8005e00 2021-03-09T10:10:38.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.901+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:38.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.901+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:38.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.901+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:38.912 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:38.912 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:38.912 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:38.912 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:38.913 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:38.913 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:38.913 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:38.913 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:38.913 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:38.914 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:38.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8005e00 2021-03-09T10:10:38.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.905+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:38.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8005e00 2021-03-09T10:10:38.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:38.905+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:10:39.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:39.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:39.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:39.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:40.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:40.295+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:40.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:40.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:40.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:40.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:40.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:40.469+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:41.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:41.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:41.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:41.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:42.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:42.468+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:42.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:42.468+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:43.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:43.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:43.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:43.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:43.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13579 cs=0 l=1 rev1=1 rx=0x7f63fc02bff0 tx=0x7f63fc052fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:43.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13579 cs=0 l=1 rev1=1 rx=0x7f63fc02bff0 tx=0x7f63fc052fd0).stop 2021-03-09T10:10:43.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:43.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:43.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:43.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:43.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:10:43.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:43.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:43.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:43.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:43.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:43.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:43.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:43.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce70 con 0x7f63d8003400 2021-03-09T10:10:43.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:43.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015260 con 0x7f63d8009130 2021-03-09T10:10:43.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:43.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:10:43.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:43.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:43.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:43.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:43.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.901+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:43.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.902+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:43.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.902+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:10:43.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.902+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:43.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.902+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:43.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.903+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:43.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.903+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:43.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.903+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:43.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:43.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:43.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:43.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:43.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:43.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:10:43.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:10:43.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:43.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:43.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015a70 con 0x7f63d8003840 2021-03-09T10:10:43.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:43.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:43.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:10:43.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:10:43.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.904+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:43.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.905+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:10:43.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.905+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a450 secure :-1 s=READY pgs=13593 cs=0 l=1 rev1=1 rx=0x7f63f00216e0 tx=0x7f63f0011360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:43.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.905+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:43.917 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:43.906+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37514 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:43.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0073f00 con 0x7f63d8003840 2021-03-09T10:10:43.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.907+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:43.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.907+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:43.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.907+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:43.919 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:43.919 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:43.919 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:43.919 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:43.920 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:43.920 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:43.920 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:43.920 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:43.920 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:43.921 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:43.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.911+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0073f00 con 0x7f63d8003840 2021-03-09T10:10:43.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.911+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:43.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.911+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007bdd0 con 0x7f63d8003840 2021-03-09T10:10:43.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:43.911+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:10:44.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:44.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:44.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:44.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:45.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:45.296+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:45.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:45.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:45.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:45.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:46.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:46.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:46.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:46.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:47.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:47.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:47.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:47.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:48.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:48.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:48.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:48.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:48.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a450 secure :-1 s=READY pgs=13593 cs=0 l=1 rev1=1 rx=0x7f63f00216e0 tx=0x7f63f0011360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:48.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a450 secure :-1 s=READY pgs=13593 cs=0 l=1 rev1=1 rx=0x7f63f00216e0 tx=0x7f63f0011360).stop 2021-03-09T10:10:48.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:48.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:48.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:48.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a450 unknown :-1 s=CLOSED pgs=13593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:48.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:10:48.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:48.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:48.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:48.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:48.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:48.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:48.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:48.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197e0 con 0x7f63d8003400 2021-03-09T10:10:48.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:48.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8009130 2021-03-09T10:10:48.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:48.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00018d0 con 0x7f63d8005e00 2021-03-09T10:10:48.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.907+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:48.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:48.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.908+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:48.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:48.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.908+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:48.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.908+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:48.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.908+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:10:48.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.908+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:48.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.908+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:48.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:48.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:48.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:48.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:48.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:48.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:48.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:48.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:48.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:10:48.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:10:48.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:48.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:48.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T10:10:48.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:48.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:48.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015a70 con 0x7f63d8005e00 2021-03-09T10:10:48.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:10:48.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:48.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13603 cs=0 l=1 rev1=1 rx=0x7f63f0013b50 tx=0x7f63f0057400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:48.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.909+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:10:48.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.910+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:48.923 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:48.912+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37552 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:48.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T10:10:48.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.913+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:48.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.913+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:48.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.913+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:48.925 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:48.925 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:48.925 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:48.925 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:48.926 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:48.926 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:48.926 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:48.926 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:48.926 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:48.927 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.915+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00212f0 con 0x7f63d8005e00 2021-03-09T10:10:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.915+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.915+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:10:48.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:48.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007c350 con 0x7f63d8005e00 2021-03-09T10:10:49.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:49.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:49.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:49.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:50.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:50.297+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:50.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:50.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:50.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:50.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:50.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:50.469+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:51.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:51.469+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:51.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:51.469+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:52.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:52.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:52.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:52.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:53.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:53.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:53.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.912+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:53.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.912+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:53.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.912+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13603 cs=0 l=1 rev1=1 rx=0x7f63f0013b50 tx=0x7f63f0057400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:53.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.912+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13603 cs=0 l=1 rev1=1 rx=0x7f63f0013b50 tx=0x7f63f0057400).stop 2021-03-09T10:10:53.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:53.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:53.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:53.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:53.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:10:53.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:53.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:53.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:53.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:53.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:53.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:53.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:53.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ece0 con 0x7f63d8003400 2021-03-09T10:10:53.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:53.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:10:53.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:53.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce70 con 0x7f63d8003840 2021-03-09T10:10:53.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:53.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b260 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:53.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:53.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:53.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:53.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:53.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:10:53.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:53.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.913+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:53.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.914+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:53.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.914+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:53.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.914+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:53.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:53.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:53.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:53.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:53.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:53.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:10:53.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:10:53.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:53.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:53.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:10:53.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:53.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:53.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b70 con 0x7f63d8003840 2021-03-09T10:10:53.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:10:53.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.915+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:53.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a400 secure :-1 s=READY pgs=13616 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc0235c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:53.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.916+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:10:53.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.916+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:53.928 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:53.918+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37592 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:53.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0133e0 con 0x7f63d8003840 2021-03-09T10:10:53.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.919+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:53.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.919+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:53.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.919+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:53.930 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:53.930 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:53.930 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:53.931 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:53.931 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:53.931 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:53.931 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:53.932 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:53.932 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:53.932 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:53.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0133e0 con 0x7f63d8003840 2021-03-09T10:10:53.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.921+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:53.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63d8003840 2021-03-09T10:10:53.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:53.923+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:10:54.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:54.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:54.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:54.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:55.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:55.298+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:10:55.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:55.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:55.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:55.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:56.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:56.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:56.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:56.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:57.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:57.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:57.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:57.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:58.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.470+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:58.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.470+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:58.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.919+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:10:58.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.919+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:10:58.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.919+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a400 secure :-1 s=READY pgs=13616 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc0235c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:10:58.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.919+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a400 secure :-1 s=READY pgs=13616 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc0235c0).stop 2021-03-09T10:10:58.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:10:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:10:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a400 unknown :-1 s=CLOSED pgs=13616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:10:58.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:10:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:10:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:10:58.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:10:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:10:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00657b0 con 0x7f63d8003400 2021-03-09T10:10:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:58.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00197e0 con 0x7f63d8009130 2021-03-09T10:10:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:10:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d670 con 0x7f63d8005e00 2021-03-09T10:10:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.920+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:10:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:10:58.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:58.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:10:58.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:10:58.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:10:58.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.921+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:10:58.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.921+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:10:58.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.921+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:10:58.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:10:58.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:10:58.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:10:58.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:10:58.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:58.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:58.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:10:58.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:10:58.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:10:58.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:10:58.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:10:58.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:58.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e4d0 con 0x7f63d8005e00 2021-03-09T10:10:58.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:10:58.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:10:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:10:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:10:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.923+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:10:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.924+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:10:58.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.924+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13637 cs=0 l=1 rev1=1 rx=0x7f63fc00bec0 tx=0x7f63fc02b590).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:10:58.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.924+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:10:58.950 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:10:58.924+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37629 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:10:58.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023400 con 0x7f63d8005e00 2021-03-09T10:10:58.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.925+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:10:58.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.925+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:10:58.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.925+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:10:58.952 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:10:58.952 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:10:58.952 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:58.952 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:10:58.953 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:10:58.953 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:10:58.953 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:10:58.953 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:10:58.953 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:10:58.954 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:10:58.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.931+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023400 con 0x7f63d8005e00 2021-03-09T10:10:58.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.931+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:10:58.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.931+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:10:58.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:58.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8005e00 2021-03-09T10:10:59.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:59.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:10:59.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:10:59.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:00.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:00.299+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:00.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:00.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:00.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:00.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:00.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:00.471+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:01.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:01.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:01.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:01.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:02.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:02.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:02.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:02.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:03.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:03.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:03.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:03.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:03.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13637 cs=0 l=1 rev1=1 rx=0x7f63fc00bec0 tx=0x7f63fc02b590).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:03.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13637 cs=0 l=1 rev1=1 rx=0x7f63fc00bec0 tx=0x7f63fc02b590).stop 2021-03-09T10:11:03.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:03.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:03.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:03.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:03.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:11:03.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:03.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:03.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:03.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:03.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:03.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:03.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:03.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a630 con 0x7f63d8003400 2021-03-09T10:11:03.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:03.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ece0 con 0x7f63d8009130 2021-03-09T10:11:03.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:03.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:11:03.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.925+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:03.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.926+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:03.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.926+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:03.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.926+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:03.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.926+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:03.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.926+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:03.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.926+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:11:03.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.926+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:03.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.926+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:03.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.927+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:03.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.927+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:03.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.927+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:03.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:03.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:03.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:03.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:03.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:03.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:11:03.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:11:03.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:03.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:03.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c9f0 con 0x7f63d8003840 2021-03-09T10:11:03.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:03.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:03.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8003840 2021-03-09T10:11:03.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:11:03.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.928+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:03.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.929+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:11:03.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.929+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:03.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021340 secure :-1 s=READY pgs=13648 cs=0 l=1 rev1=1 rx=0x7f63f004db60 tx=0x7f63f0088770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:03.941 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:03.931+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37670 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:03.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.931+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:11:03.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.931+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:03.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.931+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:03.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.931+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:03.943 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:03.943 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:03.943 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:03.944 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:03.944 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:03.944 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:03.944 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:03.944 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:03.945 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:03.945 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:03.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8003840 2021-03-09T10:11:03.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.933+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:03.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.933+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:11:03.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:03.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:11:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:04.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:04.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:05.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:05.300+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:05.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:05.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:05.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:05.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:06.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:06.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:06.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:06.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:07.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:07.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:07.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:07.471+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:08.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.471+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:08.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:08.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:08.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:08.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021340 secure :-1 s=READY pgs=13648 cs=0 l=1 rev1=1 rx=0x7f63f004db60 tx=0x7f63f0088770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:08.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021340 secure :-1 s=READY pgs=13648 cs=0 l=1 rev1=1 rx=0x7f63f004db60 tx=0x7f63f0088770).stop 2021-03-09T10:11:08.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:08.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:08.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021340 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:08.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021340 unknown :-1 s=CLOSED pgs=13648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:08.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:11:08.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.931+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:08.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:08.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:08.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:08.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:08.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:08.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:08.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00657b0 con 0x7f63d8003400 2021-03-09T10:11:08.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:08.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a10 con 0x7f63d8009130 2021-03-09T10:11:08.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:08.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d670 con 0x7f63d8005e00 2021-03-09T10:11:08.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:08.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:08.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:08.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:08.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:08.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:08.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:11:08.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:08.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.932+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:08.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:08.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:08.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:08.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:08.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:08.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:08.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:08.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:08.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:11:08.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:11:08.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:08.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:08.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00180b0 con 0x7f63d8005e00 2021-03-09T10:11:08.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:08.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:08.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c9f0 con 0x7f63d8005e00 2021-03-09T10:11:08.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:11:08.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:08.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13655 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0006680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:08.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.934+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:11:08.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.934+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:08.949 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:08.936+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37707 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:08.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e430 con 0x7f63d8005e00 2021-03-09T10:11:08.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.936+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:08.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.936+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:08.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.936+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:08.951 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:08.952 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:08.952 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:08.952 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:08.952 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:08.953 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:08.953 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:08.953 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:08.953 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:08.953 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:08.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8005e00 2021-03-09T10:11:08.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.937+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:08.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.937+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:11:08.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:08.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T10:11:09.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:09.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:09.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:09.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:10.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:10.301+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:10.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:10.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:10.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:10.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:10.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:10.472+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:11.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:11.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:11.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:11.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:12.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:12.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:12.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:12.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:13.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:13.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:13.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:13.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:13.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13655 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0006680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:13.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13655 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0006680).stop 2021-03-09T10:11:13.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:13.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:13.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:13.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:13.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:11:13.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:13.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:13.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:13.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:13.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:13.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:13.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:13.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8003400 2021-03-09T10:11:13.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:13.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:11:13.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:13.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001b10 con 0x7f63d8003840 2021-03-09T10:11:13.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:13.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:13.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:13.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:13.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.937+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:13.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.938+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:13.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.938+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:11:13.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.938+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:13.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.938+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:13.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.938+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:13.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.938+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:13.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.938+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:13.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:13.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:13.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:13.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:13.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:13.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:11:13.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:11:13.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:13.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:13.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015a70 con 0x7f63d8003840 2021-03-09T10:11:13.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:13.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:13.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00180b0 con 0x7f63d8003840 2021-03-09T10:11:13.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:11:13.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.939+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:13.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.940+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:11:13.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.940+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:13.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.940+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ebc0 secure :-1 s=READY pgs=13670 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007f5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:13.952 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:13.942+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37747 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:13.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:11:13.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.942+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:13.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.942+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:13.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.942+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:13.954 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:13.954 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:13.954 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:13.954 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:13.955 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:13.955 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:13.955 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:13.956 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:13.956 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:13.956 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:13.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.948+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:11:13.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.948+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:13.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.948+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:11:13.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:13.948+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8003840 2021-03-09T10:11:14.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:14.472+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:14.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:14.472+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:15.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:15.302+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:15.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:15.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:15.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:16.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:16.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:16.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:16.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:17.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:17.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:18.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:18.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:18.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:18.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:18.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ebc0 secure :-1 s=READY pgs=13670 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007f5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:18.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ebc0 secure :-1 s=READY pgs=13670 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007f5a0).stop 2021-03-09T10:11:18.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:18.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:18.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ebc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:18.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ebc0 unknown :-1 s=CLOSED pgs=13670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:18.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:11:18.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:18.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:18.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:18.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:18.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:18.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:18.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:18.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001910 con 0x7f63d8003400 2021-03-09T10:11:18.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:18.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00247a0 con 0x7f63d8009130 2021-03-09T10:11:18.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:18.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8005e00 2021-03-09T10:11:18.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:18.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:18.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:18.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:18.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:18.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:18.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:11:18.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:18.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.943+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:18.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.944+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:18.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.944+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:18.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.944+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:18.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:18.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:18.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:18.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:18.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:18.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:11:18.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:11:18.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:18.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:18.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b70 con 0x7f63d8005e00 2021-03-09T10:11:18.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:18.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:18.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015a70 con 0x7f63d8005e00 2021-03-09T10:11:18.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:11:18.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.945+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:18.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.946+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13683 cs=0 l=1 rev1=1 rx=0x7f63f0009a40 tx=0x7f63f0029300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:18.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.946+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:11:18.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.946+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:18.959 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:18.948+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37785 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:18.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007a430 con 0x7f63d8005e00 2021-03-09T10:11:18.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.949+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:18.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.949+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:18.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.949+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:18.960 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:18.961 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:18.961 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:18.961 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:18.961 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:18.961 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:18.962 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:18.962 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:18.962 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:18.962 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:18.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63d8005e00 2021-03-09T10:11:18.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.949+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:18.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:11:18.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:18.951+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:11:19.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:19.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:19.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:19.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:20.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:20.303+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:20.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:20.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:20.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:20.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:20.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:20.473+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:21.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:21.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:22.473+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:22.473+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:23.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:23.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:23.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:23.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:23.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13683 cs=0 l=1 rev1=1 rx=0x7f63f0009a40 tx=0x7f63f0029300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:23.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13683 cs=0 l=1 rev1=1 rx=0x7f63f0009a40 tx=0x7f63f0029300).stop 2021-03-09T10:11:23.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:23.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:23.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:23.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:23.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:11:23.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:23.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:23.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:23.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:23.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:23.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:23.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:23.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8003400 2021-03-09T10:11:23.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:23.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:11:23.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:23.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:11:23.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:23.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:23.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:11:23.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:23.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:23.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:23.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:23.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.949+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:23.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:23.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:23.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:23.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:23.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:23.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:11:23.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:11:23.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:23.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:23.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8003840 2021-03-09T10:11:23.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:23.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:23.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b70 con 0x7f63d8003840 2021-03-09T10:11:23.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:11:23.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.951+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:23.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.952+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 secure :-1 s=READY pgs=13701 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f004c340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:23.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.952+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:11:23.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.952+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:23.965 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:23.955+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37824 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:23.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:11:23.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.956+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:23.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.956+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:23.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.956+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:23.967 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:23.967 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:23.967 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:23.968 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:23.968 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:23.968 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:23.968 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:23.969 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:23.969 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:23.969 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:23.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.959+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:11:23.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.959+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:23.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.959+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:11:23.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:23.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63d8003840 2021-03-09T10:11:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:24.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:24.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:24.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:25.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:25.304+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:25.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:25.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:25.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:25.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:26.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:26.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:26.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:26.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:27.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:27.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:27.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:27.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:28.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:28.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:28.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0025e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0025e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 secure :-1 s=READY pgs=13701 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f004c340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:28.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 secure :-1 s=READY pgs=13701 cs=0 l=1 rev1=1 rx=0x7f63f0015460 tx=0x7f63f004c340).stop 2021-03-09T10:11:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0025e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:28.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 unknown :-1 s=CLOSED pgs=13701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:28.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:11:28.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:28.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:28.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:28.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:11:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8009130 2021-03-09T10:11:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:28.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00793c0 con 0x7f63d8005e00 2021-03-09T10:11:28.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:28.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:28.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:28.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:28.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.956+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:28.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.957+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:28.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.957+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:11:28.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.957+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:28.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.957+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:28.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.957+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:28.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.957+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:28.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.957+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:28.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:28.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:28.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:28.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:28.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:28.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:11:28.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:11:28.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:28.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:28.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:11:28.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:28.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:28.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e4d0 con 0x7f63d8005e00 2021-03-09T10:11:28.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:11:28.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.958+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:28.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.959+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13716 cs=0 l=1 rev1=1 rx=0x7f63e0022950 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:28.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.959+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:11:28.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.959+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:28.973 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:28.962+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37863 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:28.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03ff10 con 0x7f63d8005e00 2021-03-09T10:11:28.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.962+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:28.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.962+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:28.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.962+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:28.975 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:28.975 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:28.975 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:28.975 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:28.976 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:28.976 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:28.976 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:28.976 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:28.977 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:28.977 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:28.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63d8005e00 2021-03-09T10:11:28.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.963+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:28.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.963+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:11:28.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:28.964+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:11:29.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:29.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:29.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:29.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:30.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:30.305+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:30.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:30.474+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:30.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:30.474+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:30.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:30.474+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:31.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:31.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:31.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:32.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:32.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:32.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:32.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:33.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:33.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:33.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13716 cs=0 l=1 rev1=1 rx=0x7f63e0022950 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:33.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13716 cs=0 l=1 rev1=1 rx=0x7f63e0022950 tx=0x7f63fc04a6d0).stop 2021-03-09T10:11:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:33.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:11:33.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:33.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:33.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:33.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:33.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:33.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:33.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:33.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8003400 2021-03-09T10:11:33.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:33.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bcf0 con 0x7f63d8009130 2021-03-09T10:11:33.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:33.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:11:33.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.962+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:33.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:33.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:33.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:33.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:33.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:11:33.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:33.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:33.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:33.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:33.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:33.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.963+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:33.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:33.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:33.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:33.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:33.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:33.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:11:33.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:11:33.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:33.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:33.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b70 con 0x7f63d8003840 2021-03-09T10:11:33.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:33.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:33.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:11:33.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:11:33.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.964+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:33.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.965+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=13724 cs=0 l=1 rev1=1 rx=0x7f63fc047a90 tx=0x7f63fc06f370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:33.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.965+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:11:33.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.965+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:33.978 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:33.967+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37902 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:33.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02b800 con 0x7f63d8003840 2021-03-09T10:11:33.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.968+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:33.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.968+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:33.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.968+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:33.980 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:33.981 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:33.981 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:33.981 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:33.981 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:33.981 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:33.982 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:33.982 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:33.982 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:33.982 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:33.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02b800 con 0x7f63d8003840 2021-03-09T10:11:33.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.969+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:33.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.969+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:11:33.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:33.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63d8003840 2021-03-09T10:11:34.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:34.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:34.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:34.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:35.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:35.306+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:35.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:35.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:35.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:35.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:36.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:36.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:36.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:36.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:37.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:37.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:37.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:37.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:38.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:38.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:38.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:38.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:38.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=13724 cs=0 l=1 rev1=1 rx=0x7f63fc047a90 tx=0x7f63fc06f370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:38.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=13724 cs=0 l=1 rev1=1 rx=0x7f63fc047a90 tx=0x7f63fc06f370).stop 2021-03-09T10:11:38.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:38.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:38.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:38.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=CLOSED pgs=13724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:38.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:11:38.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:38.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:38.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:38.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:38.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:38.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:38.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:38.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T10:11:38.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:38.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a830 con 0x7f63d8009130 2021-03-09T10:11:38.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:38.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00793c0 con 0x7f63d8005e00 2021-03-09T10:11:38.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:38.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:38.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:38.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:38.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:38.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.969+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:38.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.969+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:11:38.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.969+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:38.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.969+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:38.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.969+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:38.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.969+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:38.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.969+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:38.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:38.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:38.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:38.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:38.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:38.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:11:38.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:11:38.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:38.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a700 con 0x7f63d8005e00 2021-03-09T10:11:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b70 con 0x7f63d8005e00 2021-03-09T10:11:38.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:11:38.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.971+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:38.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.972+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13746 cs=0 l=1 rev1=1 rx=0x7f63e0023ba0 tx=0x7f63fc02b670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:38.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.972+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:11:38.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.972+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:38.984 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:38.974+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37941 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:38.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc051940 con 0x7f63d8005e00 2021-03-09T10:11:38.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.975+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:38.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.975+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:38.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.975+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:38.985 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:38.985 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:38.986 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:38.986 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:38.986 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:38.986 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:38.987 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:38.987 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:38.987 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:38.987 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:38.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051b50 con 0x7f63d8005e00 2021-03-09T10:11:38.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.978+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:38.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.979+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:11:38.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:38.979+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8005e00 2021-03-09T10:11:39.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:39.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:39.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:39.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:40.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:40.307+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:40.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:40.475+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:40.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:40.475+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:40.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:40.475+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:41.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:41.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:41.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:41.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:42.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:42.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:42.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:42.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:43.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:43.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:43.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:43.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:43.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13746 cs=0 l=1 rev1=1 rx=0x7f63e0023ba0 tx=0x7f63fc02b670).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:43.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13746 cs=0 l=1 rev1=1 rx=0x7f63e0023ba0 tx=0x7f63fc02b670).stop 2021-03-09T10:11:43.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:43.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:43.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:43.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:43.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:11:43.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:43.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:43.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:43.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:43.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:43.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:43.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:43.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8003400 2021-03-09T10:11:43.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:43.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:11:43.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:43.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00793c0 con 0x7f63d8003840 2021-03-09T10:11:43.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:43.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:43.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:43.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:43.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.975+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.976+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.976+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:11:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.976+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.976+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:43.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.976+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:43.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.976+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:43.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.976+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:43.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:43.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:43.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:43.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:43.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:43.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:11:43.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:11:43.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:43.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:43.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027c60 con 0x7f63d8003840 2021-03-09T10:11:43.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:43.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:43.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a700 con 0x7f63d8003840 2021-03-09T10:11:43.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:11:43.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.977+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:43.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.978+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=13758 cs=0 l=1 rev1=1 rx=0x7f63e002cd70 tx=0x7f63fc05df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:43.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.978+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:11:43.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.978+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:43.989 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:43.980+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 37980 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:43.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052e80 con 0x7f63d8003840 2021-03-09T10:11:43.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.981+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:43.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.981+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:43.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.981+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:43.990 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:43.991 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:43.991 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:43.991 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:43.991 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:43.991 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:43.992 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:43.992 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:43.992 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:43.992 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:43.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052e80 con 0x7f63d8003840 2021-03-09T10:11:43.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.981+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:43.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.981+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:11:43.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:43.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8003840 2021-03-09T10:11:44.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:44.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:44.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:44.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:45.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:45.308+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:45.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:45.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:45.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:45.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:46.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:46.476+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:46.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:46.476+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:47.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:47.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:47.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:47.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:48.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:48.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:48.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:48.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:48.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=13758 cs=0 l=1 rev1=1 rx=0x7f63e002cd70 tx=0x7f63fc05df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:48.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=13758 cs=0 l=1 rev1=1 rx=0x7f63e002cd70 tx=0x7f63fc05df50).stop 2021-03-09T10:11:48.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:48.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:48.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:48.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=CLOSED pgs=13758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:48.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:11:48.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:48.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:48.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:48.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:48.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:48.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:48.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:48.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00235d0 con 0x7f63d8003400 2021-03-09T10:11:48.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:48.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000ad0 con 0x7f63d8009130 2021-03-09T10:11:48.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:48.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8005e00 2021-03-09T10:11:48.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:48.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:48.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:48.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:48.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:48.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:48.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:11:48.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:48.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.981+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:48.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:48.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:48.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.982+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:48.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:48.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:48.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:48.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:48.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:48.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:11:48.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:11:48.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:48.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:48.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066530 con 0x7f63d8005e00 2021-03-09T10:11:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T10:11:48.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:11:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.983+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.984+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:11:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.984+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:48.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.984+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13771 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:48.997 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:48.986+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38019 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:48.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038e80 con 0x7f63d8005e00 2021-03-09T10:11:48.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.986+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:48.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.986+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:48.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.986+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:48.998 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:48.999 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:48.999 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:48.999 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:48.999 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:48.999 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:49.000 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:49.000 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:49.000 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:49.000 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:49.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0038e80 con 0x7f63d8005e00 2021-03-09T10:11:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.987+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.987+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:11:49.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:48.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8005e00 2021-03-09T10:11:49.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:49.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:49.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:49.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:50.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:50.309+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:50.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:50.477+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:50.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:50.477+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:50.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:50.477+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:51.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:51.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:51.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:51.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:52.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:52.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:52.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:52.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:53.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:53.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:53.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:11:53.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:53.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13771 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:53.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13771 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0013b80).stop 2021-03-09T10:11:53.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:53.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:53.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:53.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13771 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:53.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:11:53.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:53.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:53.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:53.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:53.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:53.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:53.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:53.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:11:53.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:53.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00249a0 con 0x7f63d8009130 2021-03-09T10:11:53.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:53.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025be0 con 0x7f63d8003840 2021-03-09T10:11:53.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:53.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:53.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:53.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:53.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:53.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:53.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:11:53.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:53.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.987+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:53.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:53.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:53.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:53.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:53.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:53.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:53.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:53.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:53.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:11:53.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:11:53.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:53.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:53.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:11:53.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:54.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:54.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.988+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066530 con 0x7f63d8003840 2021-03-09T10:11:54.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.989+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:11:54.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.989+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:54.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.989+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13783 cs=0 l=1 rev1=1 rx=0x7f63f0028600 tx=0x7f63f005ff70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:54.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.989+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:11:54.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.989+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:54.001 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:53.992+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38058 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:54.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:11:54.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.993+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:54.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.993+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:54.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.993+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:54.003 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:54.003 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:54.003 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:54.004 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:54.004 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:54.004 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:54.004 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:54.004 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:54.005 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:54.005 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:54.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:11:54.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.995+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:54.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.995+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:11:54.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:53.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00481f0 con 0x7f63d8003840 2021-03-09T10:11:54.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:54.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:54.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:54.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:55.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:55.310+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:11:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:55.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:55.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:55.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:56.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:56.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:56.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:56.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:57.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:57.478+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:57.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:57.478+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:58.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:58.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:11:58.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:11:58.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13783 cs=0 l=1 rev1=1 rx=0x7f63f0028600 tx=0x7f63f005ff70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:11:58.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13783 cs=0 l=1 rev1=1 rx=0x7f63f0028600 tx=0x7f63f005ff70).stop 2021-03-09T10:11:58.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:58.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:11:58.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:11:58.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=CLOSED pgs=13783 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:11:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:11:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:11:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:58.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:11:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:11:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:11:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:11:58.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:59.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8009130 2021-03-09T10:11:59.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:11:59.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8005e00 2021-03-09T10:11:59.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:11:59.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:11:59.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:59.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:11:59.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:11:59.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.994+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:11:59.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.994+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:11:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.994+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:11:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.994+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:11:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.995+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:11:59.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.995+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:11:59.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.995+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:11:59.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:11:59.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:59.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:59.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:11:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:11:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:11:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:11:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:11:59.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:59.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T10:11:59.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:11:59.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:11:59.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:11:59.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:11:59.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.996+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:11:59.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13802 cs=0 l=1 rev1=1 rx=0x7f63f0023fd0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:11:59.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.997+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:11:59.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.997+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:11:59.008 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:11:58.998+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38095 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:11:59.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00884f0 con 0x7f63d8005e00 2021-03-09T10:11:59.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.999+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:11:59.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.999+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:11:59.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:58.999+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:11:59.010 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:11:59.010 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:11:59.010 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:59.010 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:11:59.011 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:11:59.011 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:11:59.011 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:11:59.011 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:11:59.011 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:11:59.012 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:11:59.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:59.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:11:59.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:59.003+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:11:59.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:59.003+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:11:59.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:59.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T10:11:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:59.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:11:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:11:59.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:00.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:00.311+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:00.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:00.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:00.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:00.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:00.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:00.479+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:01.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:01.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:01.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:02.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:02.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:02.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:02.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:03.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:03.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:04.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:04.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:04.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13802 cs=0 l=1 rev1=1 rx=0x7f63f0023fd0 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:04.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13802 cs=0 l=1 rev1=1 rx=0x7f63f0023fd0 tx=0x7f63f004db20).stop 2021-03-09T10:12:04.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:04.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:04.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:04.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13802 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:04.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:12:04.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:04.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:04.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:04.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:03.999+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:04.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:04.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:04.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:04.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T10:12:04.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:04.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00793c0 con 0x7f63d8009130 2021-03-09T10:12:04.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:04.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00249a0 con 0x7f63d8003840 2021-03-09T10:12:04.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:04.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:04.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:04.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:04.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:04.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:04.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:12:04.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:04.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:04.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:04.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:04.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.000+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:04.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:04.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:04.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:04.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:04.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:04.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:12:04.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:12:04.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:04.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:04.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b70 con 0x7f63d8003840 2021-03-09T10:12:04.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:04.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:04.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8003840 2021-03-09T10:12:04.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:12:04.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.001+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:04.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.002+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13809 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0011f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:04.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.002+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:12:04.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.002+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:04.015 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:04.005+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38136 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:04.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:12:04.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.006+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:04.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.006+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:04.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.006+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:04.017 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:04.017 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:04.017 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:04.017 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:04.018 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:04.018 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:04.018 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:04.018 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:04.018 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:04.019 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:04.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:12:04.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.008+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:04.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.008+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:12:04.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028b40 con 0x7f63d8003840 2021-03-09T10:12:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.479+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:04.479+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:05.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:05.312+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:05.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:05.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:05.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:05.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:06.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:06.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:06.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:07.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:07.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:07.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:07.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:08.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:08.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:08.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:08.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:09.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:09.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:09.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13809 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0011f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:09.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13809 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0011f50).stop 2021-03-09T10:12:09.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:09.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:09.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:09.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=CLOSED pgs=13809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:09.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:12:09.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:09.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:09.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:09.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:09.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:09.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:09.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:09.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001be30 con 0x7f63d8003400 2021-03-09T10:12:09.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:09.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009450 con 0x7f63d8009130 2021-03-09T10:12:09.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:09.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8005e00 2021-03-09T10:12:09.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:09.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:09.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:09.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:09.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.007+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:09.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:09.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.008+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:12:09.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.008+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:09.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.008+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:09.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.008+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:09.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.008+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:09.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.008+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:09.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:09.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:09.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:09.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:09.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:09.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:12:09.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:12:09.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:09.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:09.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066530 con 0x7f63d8005e00 2021-03-09T10:12:09.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:09.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:09.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b70 con 0x7f63d8005e00 2021-03-09T10:12:09.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:12:09.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:09.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13823 cs=0 l=1 rev1=1 rx=0x7f63f0027820 tx=0x7f63f007c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:09.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.010+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:12:09.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.010+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:09.022 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:09.012+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38185 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:09.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002f1d0 con 0x7f63d8005e00 2021-03-09T10:12:09.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.013+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:09.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.013+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:09.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.013+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:09.024 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:09.024 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:09.024 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:09.025 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:09.025 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:09.025 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:09.025 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:09.025 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:09.026 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:09.026 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:09.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002f1d0 con 0x7f63d8005e00 2021-03-09T10:12:09.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.013+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:09.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.013+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:12:09.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.015+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0088b60 con 0x7f63d8005e00 2021-03-09T10:12:09.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:09.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:09.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:10.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:10.313+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:10.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:10.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:10.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:10.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:10.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:10.480+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:11.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:11.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:11.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:12.480+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:12.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:12.480+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:13.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:13.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:13.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:13.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:14.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:14.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:14.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13823 cs=0 l=1 rev1=1 rx=0x7f63f0027820 tx=0x7f63f007c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:14.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13823 cs=0 l=1 rev1=1 rx=0x7f63f0027820 tx=0x7f63f007c430).stop 2021-03-09T10:12:14.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:14.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:14.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:14.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:14.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:12:14.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:14.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:14.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:14.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:14.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:14.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:14.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:14.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00158b0 con 0x7f63d8003400 2021-03-09T10:12:14.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:14.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000ad0 con 0x7f63d8009130 2021-03-09T10:12:14.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:14.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00237d0 con 0x7f63d8003840 2021-03-09T10:12:14.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:14.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:14.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:14.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.013+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:14.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.014+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:14.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.014+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:14.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.014+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:12:14.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.014+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:14.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.014+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:14.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:14.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:14.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:14.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:14.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:14.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:14.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:14.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:14.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:12:14.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:12:14.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:14.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:14.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8003840 2021-03-09T10:12:14.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:14.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:14.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027c60 con 0x7f63d8003840 2021-03-09T10:12:14.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:12:14.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.015+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:14.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.016+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13838 cs=0 l=1 rev1=1 rx=0x7f63fc06dee0 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:14.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.016+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:12:14.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.016+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:14.028 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:14.019+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38224 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:14.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc029290 con 0x7f63d8003840 2021-03-09T10:12:14.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.021+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:14.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.021+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:14.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.021+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:14.029 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:14.030 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:14.030 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:14.030 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:14.030 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:14.031 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:14.031 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:14.031 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:14.031 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:14.031 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:14.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc029290 con 0x7f63d8003840 2021-03-09T10:12:14.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.023+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:14.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.023+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:12:14.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:12:14.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:14.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:14.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:15.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:15.314+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:15.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:15.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:15.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:15.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:16.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:16.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:16.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:16.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:17.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:17.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:17.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:17.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:18.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:18.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:18.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:18.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:19.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:19.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:19.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13838 cs=0 l=1 rev1=1 rx=0x7f63fc06dee0 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:19.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13838 cs=0 l=1 rev1=1 rx=0x7f63fc06dee0 tx=0x7f63fc04a740).stop 2021-03-09T10:12:19.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:19.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:19.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:19.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=CLOSED pgs=13838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:19.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:12:19.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:19.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:19.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:19.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:19.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:19.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:19.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:19.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:12:19.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:19.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c000 con 0x7f63d8009130 2021-03-09T10:12:19.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:19.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:12:19.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:19.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:19.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:19.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:19.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.020+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:19.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.021+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:19.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.021+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:12:19.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.021+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:19.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.021+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:19.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:19.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:19.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:19.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:19.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002e050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:19.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:19.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:19.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:19.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:12:19.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:12:19.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:19.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:19.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:12:19.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:19.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:19.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T10:12:19.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:12:19.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:19.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13852 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc012870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:19.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:12:19.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.022+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:19.036 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:19.026+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38263 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:19.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052e80 con 0x7f63d8005e00 2021-03-09T10:12:19.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.027+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:19.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.027+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:19.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.027+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:19.038 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:19.038 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:19.038 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:19.038 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:19.039 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:19.040 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:19.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052e80 con 0x7f63d8005e00 2021-03-09T10:12:19.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.032+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:19.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f63d8005e00 2021-03-09T10:12:19.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.033+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:12:19.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.481+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:19.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:19.481+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:20.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:20.315+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:20.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:20.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:20.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:20.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:20.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:20.482+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:21.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:21.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:21.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:21.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:22.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:22.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:22.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:22.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:23.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:23.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:23.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:23.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:24.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.026+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:24.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.026+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:24.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.026+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13852 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc012870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:24.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.026+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13852 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc012870).stop 2021-03-09T10:12:24.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:24.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:24.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:24.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:24.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:12:24.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:24.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:24.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:24.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:24.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:24.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:24.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:24.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a4e0 con 0x7f63d8003400 2021-03-09T10:12:24.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:24.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c4f0 con 0x7f63d8009130 2021-03-09T10:12:24.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:24.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c000 con 0x7f63d8003840 2021-03-09T10:12:24.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:24.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002df60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:24.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:24.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:24.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.027+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:24.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.028+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:24.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.028+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:12:24.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.028+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:24.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.028+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:24.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.028+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:24.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.028+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:24.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.028+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:24.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:24.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002df60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:24.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:24.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:24.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:24.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:12:24.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:12:24.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:24.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:24.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b70 con 0x7f63d8003840 2021-03-09T10:12:24.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:24.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:24.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066530 con 0x7f63d8003840 2021-03-09T10:12:24.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:12:24.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.030+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:24.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.031+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13874 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:24.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.031+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:12:24.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.031+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:24.042 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:24.033+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38302 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:24.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:12:24.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.033+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:24.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.033+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:24.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.033+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:24.044 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:24.044 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:24.044 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:24.045 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:24.045 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:24.045 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:24.045 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:24.046 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:24.046 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:24.046 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:24.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:12:24.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.035+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:24.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.035+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:12:24.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:12:24.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:24.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:24.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:25.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:25.316+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:25.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:25.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:25.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:25.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:26.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:26.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:26.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:26.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:27.482+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:27.482+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:28.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:28.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:28.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:28.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:29.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:29.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:29.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13874 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:29.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 secure :-1 s=READY pgs=13874 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0013b80).stop 2021-03-09T10:12:29.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:29.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:29.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:29.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005cba0 unknown :-1 s=CLOSED pgs=13874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:29.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:12:29.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:29.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:29.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:29.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:29.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:29.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:29.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:29.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8003400 2021-03-09T10:12:29.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:29.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00150d0 con 0x7f63d8009130 2021-03-09T10:12:29.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:29.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:12:29.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:29.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:29.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:29.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:29.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:29.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.034+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:29.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.034+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:12:29.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.034+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:29.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.034+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:29.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:29.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:29.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:29.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:29.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:29.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:29.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:29.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:29.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:12:29.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:12:29.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:29.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:29.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a120 con 0x7f63d8005e00 2021-03-09T10:12:29.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:29.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:29.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b70 con 0x7f63d8005e00 2021-03-09T10:12:29.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:12:29.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.035+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:29.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.036+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:12:29.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.036+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:29.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.036+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13882 cs=0 l=1 rev1=1 rx=0x7f63f0013420 tx=0x7f63f0041f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:29.048 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:29.039+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38341 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:29.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:12:29.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.040+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:29.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.040+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:29.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.040+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:29.049 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:29.050 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:29.050 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:29.050 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:29.050 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:29.051 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:29.051 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:29.051 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:29.051 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:29.051 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:29.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:12:29.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.044+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:29.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.044+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:12:29.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8005e00 2021-03-09T10:12:29.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:29.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:29.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:30.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:30.317+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:30.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:30.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:30.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:30.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:30.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:30.483+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:31.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:31.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:31.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:31.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:32.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:32.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:32.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:32.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:33.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:33.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:33.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:33.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:34.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:12:34.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:34.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13882 cs=0 l=1 rev1=1 rx=0x7f63f0013420 tx=0x7f63f0041f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:34.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13882 cs=0 l=1 rev1=1 rx=0x7f63f0013420 tx=0x7f63f0041f30).stop 2021-03-09T10:12:34.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:34.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:34.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:34.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:34.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:12:34.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:34.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:34.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:34.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:34.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:34.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:34.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:34.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:12:34.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:34.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010c50 con 0x7f63d8009130 2021-03-09T10:12:34.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:34.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065430 con 0x7f63d8003840 2021-03-09T10:12:34.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:34.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:34.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:34.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:34.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:34.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:34.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:12:34.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:34.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.040+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:34.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.041+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:34.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.041+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:34.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.041+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:34.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.042+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:34.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.042+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:34.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:34.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001b120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:34.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001b120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:34.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:12:34.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:12:34.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:34.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:34.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079370 con 0x7f63d8003840 2021-03-09T10:12:34.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:34.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:34.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a120 con 0x7f63d8003840 2021-03-09T10:12:34.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:12:34.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.043+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:34.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.044+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:12:34.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.044+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 secure :-1 s=READY pgs=13899 cs=0 l=1 rev1=1 rx=0x7f63f0031490 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:34.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.044+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:34.055 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:34.046+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38380 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:34.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:12:34.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.052+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:34.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.052+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:34.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.052+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:34.056 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:34.057 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:34.057 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:34.057 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:34.057 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:34.057 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:34.058 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:34.058 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:34.058 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:34.058 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:34.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0011d80 con 0x7f63d8003840 2021-03-09T10:12:34.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.052+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:34.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.052+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:12:34.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:12:34.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:34.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:34.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:35.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:35.317+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:35.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:35.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:35.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:35.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:36.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:36.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:36.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:36.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:37.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:37.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:37.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:37.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:38.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:38.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:38.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:38.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:39.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.045+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:39.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.045+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:39.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.045+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 secure :-1 s=READY pgs=13899 cs=0 l=1 rev1=1 rx=0x7f63f0031490 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:39.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.045+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 secure :-1 s=READY pgs=13899 cs=0 l=1 rev1=1 rx=0x7f63f0031490 tx=0x7f63f004db20).stop 2021-03-09T10:12:39.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.045+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:39.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.045+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:39.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:39.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 unknown :-1 s=CLOSED pgs=13899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:39.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.045+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:12:39.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:39.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:39.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:39.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:39.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:39.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:39.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:39.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:12:39.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:39.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8009130 2021-03-09T10:12:39.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:39.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068620 con 0x7f63d8005e00 2021-03-09T10:12:39.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:39.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:39.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:39.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:39.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:39.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:39.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:12:39.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:39.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.046+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:39.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.047+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:39.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.047+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:39.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.047+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:39.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:39.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:39.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:39.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:39.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:39.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:12:39.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:12:39.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:39.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:39.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b70 con 0x7f63d8005e00 2021-03-09T10:12:39.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:39.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:39.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079370 con 0x7f63d8005e00 2021-03-09T10:12:39.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:12:39.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.048+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:39.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.049+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13912 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f005fe90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:39.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.049+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:12:39.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.049+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:39.061 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:39.051+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38419 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:39.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:12:39.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.053+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:39.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.053+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:39.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.053+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:39.062 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:39.063 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:39.063 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:39.063 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:39.063 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:39.063 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:39.064 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:39.064 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:39.064 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:39.064 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:39.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.056+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:12:39.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.056+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:39.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.056+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:12:39.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0073340 con 0x7f63d8005e00 2021-03-09T10:12:39.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:39.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:39.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:40.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:40.318+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:40.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:40.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:40.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:40.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:40.483+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:41.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:41.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:41.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:41.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:42.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:42.483+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:42.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:42.483+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:43.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:43.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:43.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:43.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:44.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:44.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:44.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13912 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f005fe90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:44.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13912 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f005fe90).stop 2021-03-09T10:12:44.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:44.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:44.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:44.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:44.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:12:44.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:44.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:44.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:44.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:44.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:44.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:44.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:44.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001be30 con 0x7f63d8003400 2021-03-09T10:12:44.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:44.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012a70 con 0x7f63d8009130 2021-03-09T10:12:44.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:44.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029640 con 0x7f63d8003840 2021-03-09T10:12:44.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:44.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:44.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:44.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:44.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:44.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:44.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:12:44.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:44.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:44.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:44.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:44.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:44.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:44.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:44.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:44.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:44.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:44.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:12:44.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:12:44.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:44.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:44.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079de0 con 0x7f63d8003840 2021-03-09T10:12:44.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:44.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:44.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:12:44.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:12:44.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.053+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:44.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.054+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:12:44.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.054+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 secure :-1 s=READY pgs=13919 cs=0 l=1 rev1=1 rx=0x7f63e0025490 tx=0x7f63fc06f370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:44.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.054+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:44.075 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:44.058+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38458 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:44.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:12:44.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.058+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:44.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.058+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:44.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.058+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:44.077 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:44.078 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:44.078 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:44.078 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:44.079 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:44.079 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:44.079 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:44.080 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:44.080 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:44.080 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:44.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:12:44.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.058+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:44.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.058+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:12:44.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:12:44.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:44.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:44.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:45.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:45.319+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:45.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:45.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:45.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:45.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:46.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:46.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:46.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:46.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:47.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:47.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:47.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:47.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:48.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:48.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:48.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:48.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:49.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:12:49.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:49.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 secure :-1 s=READY pgs=13919 cs=0 l=1 rev1=1 rx=0x7f63e0025490 tx=0x7f63fc06f370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:49.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 secure :-1 s=READY pgs=13919 cs=0 l=1 rev1=1 rx=0x7f63e0025490 tx=0x7f63fc06f370).stop 2021-03-09T10:12:49.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:49.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:49.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:49.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 unknown :-1 s=CLOSED pgs=13919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:49.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:12:49.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:49.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:49.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:49.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:49.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:49.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:49.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:49.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00116d0 con 0x7f63d8003400 2021-03-09T10:12:49.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:49.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b3d0 con 0x7f63d8009130 2021-03-09T10:12:49.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:49.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:12:49.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.058+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:49.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:49.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:49.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:49.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:49.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.060+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:49.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.060+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:12:49.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.060+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:49.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.060+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:49.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.060+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:49.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.060+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:49.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.060+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:49.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:49.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:49.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:49.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:49.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:49.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:12:49.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:12:49.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:49.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:49.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00018b0 con 0x7f63d8005e00 2021-03-09T10:12:49.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:49.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:49.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079de0 con 0x7f63d8005e00 2021-03-09T10:12:49.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:12:49.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.061+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:49.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.062+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:12:49.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.062+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:49.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.062+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13937 cs=0 l=1 rev1=1 rx=0x7f63e000d830 tx=0x7f63fc03ff10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:49.075 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:49.064+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38497 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:49.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04bcd0 con 0x7f63d8005e00 2021-03-09T10:12:49.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.065+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:49.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.065+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:49.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.065+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:49.077 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:49.077 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:49.077 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:49.077 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:49.078 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:49.078 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:49.078 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:49.078 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:49.079 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:49.079 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:49.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04bcd0 con 0x7f63d8005e00 2021-03-09T10:12:49.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.068+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:49.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.068+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:12:49.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:12:49.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:49.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:49.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:50.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:50.320+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:50.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:50.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:50.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:50.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:50.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:50.484+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:51.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:51.484+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:51.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:51.484+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:52.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:52.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:52.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:52.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:53.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:53.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:53.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:53.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:54.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:54.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:54.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13937 cs=0 l=1 rev1=1 rx=0x7f63e000d830 tx=0x7f63fc03ff10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:54.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13937 cs=0 l=1 rev1=1 rx=0x7f63e000d830 tx=0x7f63fc03ff10).stop 2021-03-09T10:12:54.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:54.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:54.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:54.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13937 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:54.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:12:54.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:54.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:54.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:54.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:54.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:54.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:54.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:54.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T10:12:54.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:54.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:12:54.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:54.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021c60 con 0x7f63d8003840 2021-03-09T10:12:54.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:54.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:54.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:54.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:54.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.065+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:54.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.066+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:54.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.066+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:12:54.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.066+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:54.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.066+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:54.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.066+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:54.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.066+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:54.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.066+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:54.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:54.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:54.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:54.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:54.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:54.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:12:54.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:12:54.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:54.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:54.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022330 con 0x7f63d8003840 2021-03-09T10:12:54.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:54.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:54.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b70 con 0x7f63d8003840 2021-03-09T10:12:54.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:12:54.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.067+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:54.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.068+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 secure :-1 s=READY pgs=13954 cs=0 l=1 rev1=1 rx=0x7f63f004d230 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:54.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.068+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:12:54.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.068+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:54.081 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:54.071+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38536 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:54.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:12:54.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.072+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:54.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.072+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:54.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.072+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:54.083 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:54.083 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:54.083 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:54.083 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:54.083 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:54.084 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:54.085 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0048e60 con 0x7f63d8003840 2021-03-09T10:12:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.072+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:54.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.072+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:12:54.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.074+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0011d80 con 0x7f63d8003840 2021-03-09T10:12:54.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:54.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:54.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:55.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:55.321+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:12:55.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:55.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:55.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:55.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:56.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:56.485+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:56.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:56.485+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:57.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:57.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:57.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:57.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:58.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:58.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:58.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:58.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:59.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:12:59.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:12:59.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 secure :-1 s=READY pgs=13954 cs=0 l=1 rev1=1 rx=0x7f63f004d230 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:12:59.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 secure :-1 s=READY pgs=13954 cs=0 l=1 rev1=1 rx=0x7f63f004d230 tx=0x7f63f004dd90).stop 2021-03-09T10:12:59.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:59.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:12:59.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:12:59.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017080 unknown :-1 s=CLOSED pgs=13954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:12:59.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:12:59.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:59.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.071+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:12:59.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:59.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:12:59.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:12:59.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:12:59.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:59.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a820 con 0x7f63d8003400 2021-03-09T10:12:59.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:59.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:12:59.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:12:59.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T10:12:59.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:12:59.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:12:59.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:59.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:12:59.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:12:59.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:12:59.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:12:59.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:12:59.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:12:59.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:12:59.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:12:59.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.072+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:12:59.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:12:59.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:59.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:59.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:12:59.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:12:59.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:12:59.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:12:59.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:12:59.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:59.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015860 con 0x7f63d8005e00 2021-03-09T10:12:59.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:12:59.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:12:59.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022330 con 0x7f63d8005e00 2021-03-09T10:12:59.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.074+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:12:59.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.074+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:12:59.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13963 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f006b1e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:12:59.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.074+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:12:59.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.074+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:12:59.087 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:12:59.078+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38575 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:12:59.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:12:59.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.078+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:12:59.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.078+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:12:59.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.078+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:12:59.089 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:12:59.090 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:12:59.090 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:59.090 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:12:59.090 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:12:59.091 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:12:59.091 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:12:59.091 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:12:59.091 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:12:59.091 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:12:59.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8005e00 2021-03-09T10:12:59.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.078+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:12:59.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.078+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:12:59.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007c350 con 0x7f63d8005e00 2021-03-09T10:12:59.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:12:59.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:12:59.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:00.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:00.322+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:00.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:00.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:00.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:00.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:00.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:00.486+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:01.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:01.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:01.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:02.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:02.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:02.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:02.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:03.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:03.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:03.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:03.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:04.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13963 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f006b1e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:04.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13963 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f006b1e0).stop 2021-03-09T10:13:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:04.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:13:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:04.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:04.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:04.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:04.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:04.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012a70 con 0x7f63d8003400 2021-03-09T10:13:04.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c7d0 con 0x7f63d8009130 2021-03-09T10:13:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003840 2021-03-09T10:13:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:04.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:04.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:04.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:04.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:04.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:04.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:13:04.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:04.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.078+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:04.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:04.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:04.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:13:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:13:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:13:04.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00018b0 con 0x7f63d8003840 2021-03-09T10:13:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:13:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:04.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=13973 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc051e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:13:04.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.079+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:04.093 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:04.085+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38614 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.085+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:13:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.085+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:04.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.086+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.086+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:04.095 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:04.096 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:04.097 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:04.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:13:04.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.086+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:04.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.086+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:13:04.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065750 con 0x7f63d8003840 2021-03-09T10:13:04.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.486+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:04.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:04.486+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:05.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:05.323+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:05.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:05.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:05.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:05.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:06.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:06.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:07.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:07.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:07.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:07.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:08.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:08.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:08.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:09.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=13973 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc051e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:09.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=13973 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc051e90).stop 2021-03-09T10:13:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:09.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:09.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=13973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:09.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:13:09.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:09.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:09.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:09.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:13:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:13:09.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065d50 con 0x7f63d8005e00 2021-03-09T10:13:09.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:09.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:09.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.086+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:09.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.087+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:09.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.087+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:13:09.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.087+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:09.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.087+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:09.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:09.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:09.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:09.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:09.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:09.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:09.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:09.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:09.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:13:09.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:13:09.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:09.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:09.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001ade0 con 0x7f63d8005e00 2021-03-09T10:13:09.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:09.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:09.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:13:09.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:13:09.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.088+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:09.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.089+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13991 cs=0 l=1 rev1=1 rx=0x7f63fc06f340 tx=0x7f63fc0657c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:09.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.089+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:13:09.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.089+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:09.104 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:09.092+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38653 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:09.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03dc60 con 0x7f63d8005e00 2021-03-09T10:13:09.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.093+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:09.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.093+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:09.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.093+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:09.106 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:09.106 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:09.106 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:09.107 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:09.107 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:09.107 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:09.107 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:09.107 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:09.108 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:09.108 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:09.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.096+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05b170 con 0x7f63d8005e00 2021-03-09T10:13:09.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.096+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:09.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.096+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:13:09.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.096+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03dc60 con 0x7f63d8005e00 2021-03-09T10:13:09.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:09.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:09.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:10.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:10.324+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:10.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:10.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:10.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:10.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:10.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:10.487+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:11.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:11.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:11.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:11.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:12.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:12.487+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:12.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:12.487+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:13.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:13.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:13.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:13.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:14.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:14.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:14.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13991 cs=0 l=1 rev1=1 rx=0x7f63fc06f340 tx=0x7f63fc0657c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:14.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=13991 cs=0 l=1 rev1=1 rx=0x7f63fc06f340 tx=0x7f63fc0657c0).stop 2021-03-09T10:13:14.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:14.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:14.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:14.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=13991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:14.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:13:14.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:14.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:14.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:14.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:14.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:14.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:14.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:14.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:13:14.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:14.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024a40 con 0x7f63d8009130 2021-03-09T10:13:14.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:14.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc60 con 0x7f63d8003840 2021-03-09T10:13:14.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:14.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:14.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:14.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:14.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.093+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:14.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.094+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:14.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.094+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:13:14.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.094+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:14.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.094+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:14.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.094+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:14.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.094+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:14.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.094+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:14.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.095+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:14.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.095+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:14.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.095+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:14.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.095+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:14.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.095+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:14.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:13:14.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:13:14.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:14.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:14.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079de0 con 0x7f63d8003840 2021-03-09T10:13:14.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:14.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:14.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001ade0 con 0x7f63d8003840 2021-03-09T10:13:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:13:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.096+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 secure :-1 s=READY pgs=14007 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc078e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:14.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.097+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:13:14.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.097+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:14.110 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:14.099+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38692 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:14.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc05de90 con 0x7f63d8003840 2021-03-09T10:13:14.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.100+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:14.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.100+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:14.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.100+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:14.112 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:14.112 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:14.112 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:14.113 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:14.113 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:14.114 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:14.114 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:14.115 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:14.115 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:14.115 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:14.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63d8003840 2021-03-09T10:13:14.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.100+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:14.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.100+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:13:14.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63d8003840 2021-03-09T10:13:14.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:14.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:14.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:15.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:15.325+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:15.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:15.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:15.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:15.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:16.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:16.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:16.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:16.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:17.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:17.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:17.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:17.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:18.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:18.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:18.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:18.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:19.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:19.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:19.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 secure :-1 s=READY pgs=14007 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc078e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:19.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 secure :-1 s=READY pgs=14007 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc078e80).stop 2021-03-09T10:13:19.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:19.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:19.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:19.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 unknown :-1 s=CLOSED pgs=14007 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:19.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:13:19.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:19.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:19.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:19.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:19.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:19.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:19.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:19.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63d8003400 2021-03-09T10:13:19.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:19.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00116d0 con 0x7f63d8009130 2021-03-09T10:13:19.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:19.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00194d0 con 0x7f63d8005e00 2021-03-09T10:13:19.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:19.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:19.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:19.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:19.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.099+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:19.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:19.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:13:19.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:19.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:19.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:19.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:19.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:19.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:19.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002de00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:19.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002de00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:19.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:19.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:19.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:13:19.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:13:19.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:19.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:19.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:13:19.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:19.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:19.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015860 con 0x7f63d8005e00 2021-03-09T10:13:19.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:13:19.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.100+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:19.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.101+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14015 cs=0 l=1 rev1=1 rx=0x7f63e000d510 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:19.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.101+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:13:19.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.101+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:19.115 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:19.105+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38731 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:19.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:13:19.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.106+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:19.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.106+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:19.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.106+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:19.117 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:19.117 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:19.118 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:19.118 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:19.118 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:19.118 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:19.118 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:19.119 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:19.119 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:19.119 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:19.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.106+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:13:19.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.106+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:19.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.106+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:13:19.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.107+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007c350 con 0x7f63d8005e00 2021-03-09T10:13:19.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:19.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:19.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:20.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:20.326+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:20.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:20.488+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:20.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:20.488+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:20.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:20.488+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:21.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:21.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:21.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:21.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:22.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:22.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:22.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:22.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:23.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:23.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:23.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:23.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:24.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:13:24.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:24.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14015 cs=0 l=1 rev1=1 rx=0x7f63e000d510 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:24.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14015 cs=0 l=1 rev1=1 rx=0x7f63e000d510 tx=0x7f63f000df50).stop 2021-03-09T10:13:24.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:24.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:24.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:24.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:24.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:13:24.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:24.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:24.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:24.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:24.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:24.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:24.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:24.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c3a0 con 0x7f63d8003400 2021-03-09T10:13:24.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:24.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:13:24.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:24.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029f90 con 0x7f63d8003840 2021-03-09T10:13:24.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.105+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:24.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:24.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:24.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:24.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:13:24.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:24.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:24.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:24.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:24.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:24.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:24.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:24.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:24.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:24.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:24.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001afa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:24.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001afa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:24.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:13:24.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:13:24.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:24.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:24.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.106+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e0a0 con 0x7f63d8003840 2021-03-09T10:13:24.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.107+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:24.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.107+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:24.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.107+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079de0 con 0x7f63d8003840 2021-03-09T10:13:24.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.107+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:13:24.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.107+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:24.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.107+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:13:24.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.107+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:24.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.107+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015660 secure :-1 s=READY pgs=14029 cs=0 l=1 rev1=1 rx=0x7f63e0007070 tx=0x7f63fc051f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:24.121 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:24.112+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38770 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:24.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc008c50 con 0x7f63d8003840 2021-03-09T10:13:24.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.112+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:24.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.112+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:24.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.112+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:24.122 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:24.123 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:24.123 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:24.123 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:24.123 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:24.123 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:24.124 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:24.124 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:24.124 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:24.124 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:24.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc008c50 con 0x7f63d8003840 2021-03-09T10:13:24.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.112+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:24.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.112+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:13:24.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.113+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc05dd90 con 0x7f63d8003840 2021-03-09T10:13:24.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:24.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:24.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:25.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:25.327+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:25.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:25.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:25.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:25.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:26.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:26.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:26.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:26.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:27.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:27.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:27.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:28.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:28.489+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:28.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:28.489+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:29.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:29.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:29.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015660 secure :-1 s=READY pgs=14029 cs=0 l=1 rev1=1 rx=0x7f63e0007070 tx=0x7f63fc051f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:29.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015660 secure :-1 s=READY pgs=14029 cs=0 l=1 rev1=1 rx=0x7f63e0007070 tx=0x7f63fc051f00).stop 2021-03-09T10:13:29.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:29.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:29.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:29.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015660 unknown :-1 s=CLOSED pgs=14029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:29.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:13:29.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:29.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:29.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:29.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:29.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:29.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:29.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:29.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e9d0 con 0x7f63d8003400 2021-03-09T10:13:29.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:29.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00194d0 con 0x7f63d8009130 2021-03-09T10:13:29.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:29.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:13:29.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:29.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:29.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:29.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:29.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:29.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:29.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:13:29.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:29.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:29.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:29.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:29.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.113+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:29.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:29.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:29.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:29.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:29.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:29.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:13:29.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:13:29.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:29.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:29.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079de0 con 0x7f63d8005e00 2021-03-09T10:13:29.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:29.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:29.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:13:29.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:13:29.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:29.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14042 cs=0 l=1 rev1=1 rx=0x7f63f007ffc0 tx=0x7f63f004d110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:29.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:13:29.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.114+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:29.127 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:29.118+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38809 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:29.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007fe00 con 0x7f63d8005e00 2021-03-09T10:13:29.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.118+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:29.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.118+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:29.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.118+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:29.129 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:29.129 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:29.129 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:29.129 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:29.130 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:29.130 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:29.130 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:29.130 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:29.130 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:29.131 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:29.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007fe00 con 0x7f63d8005e00 2021-03-09T10:13:29.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.118+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:29.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.118+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:13:29.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0088df0 con 0x7f63d8005e00 2021-03-09T10:13:29.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:29.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:29.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:30.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:30.328+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:30.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:30.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:30.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:30.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:30.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:30.490+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:31.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:31.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:31.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:31.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:32.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:32.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:33.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:33.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:33.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:33.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:34.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:34.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:34.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14042 cs=0 l=1 rev1=1 rx=0x7f63f007ffc0 tx=0x7f63f004d110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:34.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14042 cs=0 l=1 rev1=1 rx=0x7f63f007ffc0 tx=0x7f63f004d110).stop 2021-03-09T10:13:34.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:34.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:34.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:34.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:34.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:13:34.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:34.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:34.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:34.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:34.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00178a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:34.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:34.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:34.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009490 con 0x7f63d8003400 2021-03-09T10:13:34.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:34.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024a40 con 0x7f63d8009130 2021-03-09T10:13:34.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:34.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63d8003840 2021-03-09T10:13:34.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:34.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:34.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.118+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:34.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.119+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:34.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.119+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:34.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.119+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00178a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:34.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.119+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:13:34.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.119+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:34.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.119+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:34.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.120+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:34.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.120+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:34.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.120+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:34.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.121+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:34.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.121+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:34.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.121+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:34.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.121+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:34.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.121+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:34.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.121+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:13:34.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.121+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:13:34.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.121+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:34.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:34.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f63d8003840 2021-03-09T10:13:34.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:34.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:34.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079de0 con 0x7f63d8003840 2021-03-09T10:13:34.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:13:34.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:34.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00178a0 secure :-1 s=READY pgs=14064 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63f0039f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:34.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:13:34.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.122+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:34.135 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:34.124+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38848 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027430 con 0x7f63d8003840 2021-03-09T10:13:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.125+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.125+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.125+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:34.136 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:34.137 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:34.137 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:34.137 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:34.137 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:34.138 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:34.138 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:34.138 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:34.138 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:34.139 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:34.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:13:34.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.128+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:34.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.128+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:13:34.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:13:34.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:34.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:34.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:35.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:35.329+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:35.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:35.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:35.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:35.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:36.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:36.490+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:36.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:36.490+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:37.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:37.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:37.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:37.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:38.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:38.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:38.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:38.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:39.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.124+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00178a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:39.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.124+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00178a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:39.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.124+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00178a0 secure :-1 s=READY pgs=14064 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63f0039f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:39.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.124+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00178a0 secure :-1 s=READY pgs=14064 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63f0039f90).stop 2021-03-09T10:13:39.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.124+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:39.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.124+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:39.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.124+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00178a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:39.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.124+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00178a0 unknown :-1 s=CLOSED pgs=14064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:39.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.124+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:13:39.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:39.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:39.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:39.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c3a0 con 0x7f63d8003400 2021-03-09T10:13:39.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:39.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:13:39.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:39.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079170 con 0x7f63d8005e00 2021-03-09T10:13:39.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:39.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:39.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:39.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:39.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:13:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:39.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.125+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:39.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:39.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:39.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:39.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:13:39.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:13:39.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:39.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:39.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:13:39.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:39.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:39.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f63d8005e00 2021-03-09T10:13:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:13:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:13:39.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:39.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14071 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f006be00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:39.139 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:39.130+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38887 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:39.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f4f0 con 0x7f63d8005e00 2021-03-09T10:13:39.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.130+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:39.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.130+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:39.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.130+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:39.141 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:39.141 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:39.141 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:39.141 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:39.142 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:39.142 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:39.142 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:39.143 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:39.143 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:39.143 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:39.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00481f0 con 0x7f63d8005e00 2021-03-09T10:13:39.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.131+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:39.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.131+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:13:39.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.132+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8005e00 2021-03-09T10:13:39.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:39.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:39.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:40.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:40.330+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:40.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:40.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:40.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:40.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:40.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:40.491+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:41.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:41.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:41.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:41.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:42.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:42.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:42.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:43.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:43.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:43.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:43.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:44.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:44.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14071 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f006be00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:44.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14071 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f006be00).stop 2021-03-09T10:13:44.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:44.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:44.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:44.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14071 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:44.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:13:44.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:44.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:44.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:44.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:44.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:44.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:44.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:44.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e8e0 con 0x7f63d8003400 2021-03-09T10:13:44.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:44.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8009130 2021-03-09T10:13:44.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:44.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:13:44.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:44.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:44.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.131+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:44.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.132+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:44.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.132+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:44.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.132+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:44.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.132+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:13:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.132+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.132+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:44.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:44.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:44.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:44.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:44.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:44.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:44.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:13:44.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:13:44.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:44.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:44.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0023a40 con 0x7f63d8003840 2021-03-09T10:13:44.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:44.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:44.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:13:44.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:13:44.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.133+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:44.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.134+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:13:44.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.134+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b880 secure :-1 s=READY pgs=14086 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f003e260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:44.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.134+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:44.146 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:44.137+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38926 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:44.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8003840 2021-03-09T10:13:44.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.138+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:44.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.138+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:44.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.138+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:44.148 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:44.148 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:44.148 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:44.149 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:44.149 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:44.149 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:44.149 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:44.149 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:44.150 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:44.150 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:44.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00481f0 con 0x7f63d8003840 2021-03-09T10:13:44.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.141+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:44.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.141+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:13:44.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8003840 2021-03-09T10:13:44.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:44.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:44.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:45.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:45.331+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:45.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:45.491+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:45.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:45.491+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:46.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:46.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:46.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:46.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:47.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:47.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:47.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:47.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:48.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:48.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:48.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:49.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b880 secure :-1 s=READY pgs=14086 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f003e260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:49.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b880 secure :-1 s=READY pgs=14086 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f003e260).stop 2021-03-09T10:13:49.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:49.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:49.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007b880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:49.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b880 unknown :-1 s=CLOSED pgs=14086 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:49.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:13:49.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:49.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:49.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:49.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:49.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:49.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:49.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:49.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079170 con 0x7f63d8003400 2021-03-09T10:13:49.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:49.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c3a0 con 0x7f63d8009130 2021-03-09T10:13:49.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:49.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:13:49.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:49.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:49.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:49.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.138+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:49.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.138+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:49.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.138+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:49.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.138+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:13:49.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.138+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:49.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.138+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.139+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.139+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.139+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:49.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:49.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:49.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001af70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:49.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001af70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:49.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:13:49.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:13:49.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:49.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:49.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002c50 con 0x7f63d8005e00 2021-03-09T10:13:49.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:49.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:49.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0023a40 con 0x7f63d8005e00 2021-03-09T10:13:49.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:13:49.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14100 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f005ff70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:13:49.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.140+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:49.152 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:49.142+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 38965 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041690 con 0x7f63d8005e00 2021-03-09T10:13:49.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.143+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.143+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.143+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:49.154 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:49.155 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:49.155 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:49.155 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:49.155 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:49.156 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:49.156 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:49.156 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:49.156 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:49.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8005e00 2021-03-09T10:13:49.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.144+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:49.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.144+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:13:49.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.145+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007fdc0 con 0x7f63d8005e00 2021-03-09T10:13:49.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:49.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:49.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:50.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:50.332+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:50.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:50.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:50.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:50.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:50.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:50.492+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:51.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:51.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:51.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:51.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:52.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:52.492+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:52.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:52.492+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:53.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:53.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:53.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:53.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:54.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:13:54.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:54.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14100 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f005ff70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:54.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14100 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f005ff70).stop 2021-03-09T10:13:54.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:54.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:54.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:54.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:54.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:13:54.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:54.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:54.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:54.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:54.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:54.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:54.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:54.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a4e0 con 0x7f63d8003400 2021-03-09T10:13:54.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:54.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8009130 2021-03-09T10:13:54.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:54.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63d8003840 2021-03-09T10:13:54.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:54.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ade0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:54.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:54.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:54.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:54.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:54.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:13:54.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:54.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.143+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:54.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:54.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:54.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:54.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:54.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:54.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:54.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ade0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:54.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:54.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:13:54.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:13:54.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:54.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:54.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8003840 2021-03-09T10:13:54.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:54.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:54.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002c50 con 0x7f63d8003840 2021-03-09T10:13:54.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:13:54.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.144+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:54.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.145+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022580 secure :-1 s=READY pgs=14111 cs=0 l=1 rev1=1 rx=0x7f63f0041a80 tx=0x7f63f0006c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:54.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.145+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:13:54.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.145+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:54.158 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:54.149+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39004 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:54.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:13:54.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.149+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:54.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.149+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:54.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.149+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:54.159 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:54.160 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:54.160 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:54.160 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:54.160 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:54.160 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:54.161 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:54.161 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:54.161 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:54.161 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:54.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.149+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:13:54.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.149+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:54.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.149+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:13:54.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:13:54.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:54.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:54.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:55.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:55.333+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:13:55.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:55.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:55.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:55.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:56.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:56.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:56.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:56.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:57.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:57.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:57.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:58.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:58.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:58.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:58.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:59.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.149+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:13:59.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.149+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:13:59.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.149+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022580 secure :-1 s=READY pgs=14111 cs=0 l=1 rev1=1 rx=0x7f63f0041a80 tx=0x7f63f0006c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:13:59.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.149+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022580 secure :-1 s=READY pgs=14111 cs=0 l=1 rev1=1 rx=0x7f63f0041a80 tx=0x7f63f0006c30).stop 2021-03-09T10:13:59.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:59.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:13:59.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:13:59.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022580 unknown :-1 s=CLOSED pgs=14111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:13:59.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:13:59.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:59.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:13:59.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:59.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:13:59.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:13:59.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:13:59.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:59.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c3a0 con 0x7f63d8003400 2021-03-09T10:13:59.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:59.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8009130 2021-03-09T10:13:59.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:13:59.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:13:59.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:13:59.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ade0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:13:59.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:13:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.150+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:13:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.151+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:13:59.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.151+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:13:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.151+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:13:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.151+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:13:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.151+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:13:59.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.151+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:13:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.151+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:13:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:13:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:59.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ade0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:13:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:13:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:13:59.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:13:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:13:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:13:59.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:13:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:13:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8005e00 2021-03-09T10:13:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:13:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:13:59.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:13:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14127 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:13:59.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.152+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:13:59.166 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:13:59.155+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39044 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:13:59.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8005e00 2021-03-09T10:13:59.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.155+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:13:59.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.155+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:13:59.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.155+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:13:59.168 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:13:59.169 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:13:59.169 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:59.169 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:13:59.169 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:13:59.170 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:13:59.170 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:13:59.170 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:13:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:13:59.171 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:13:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:13:59.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.155+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:13:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.155+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:13:59.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:13:59.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:13:59.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:13:59.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:00.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:00.334+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:00.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:00.493+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:00.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:00.493+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:00.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:00.493+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:01.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:01.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:01.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:02.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:02.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:02.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:02.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:03.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:03.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:03.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:03.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:04.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:04.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:04.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14127 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:04.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14127 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f004db20).stop 2021-03-09T10:14:04.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:04.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:04.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:04.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:04.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:14:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:04.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079170 con 0x7f63d8003400 2021-03-09T10:14:04.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:04.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63d8009130 2021-03-09T10:14:04.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:04.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:14:04.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.156+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:04.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.157+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:04.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.157+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ade0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:04.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.157+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:04.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.157+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:04.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.157+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.157+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:14:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.157+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:04.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.157+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:04.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.158+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:04.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.158+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:04.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.158+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:04.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:04.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:04.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:04.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ade0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:04.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ade0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:14:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:14:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:04.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e980 con 0x7f63d8003840 2021-03-09T10:14:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:14:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:14:04.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:14:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021f50 secure :-1 s=READY pgs=14140 cs=0 l=1 rev1=1 rx=0x7f63f0006c00 tx=0x7f63f0055290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:04.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.159+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:04.171 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:04.161+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39082 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:04.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:14:04.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.162+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:04.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.162+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:04.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.162+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:04.172 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:04.173 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:04.174 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:04.174 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:04.174 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:04.174 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:04.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:14:04.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.166+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:04.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.167+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:14:04.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:14:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:04.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:04.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:05.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:05.335+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:05.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:05.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:05.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:05.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:06.494+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:06.494+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:07.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:07.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:07.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:07.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:08.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:08.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:08.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:08.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:09.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:09.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:09.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021f50 secure :-1 s=READY pgs=14140 cs=0 l=1 rev1=1 rx=0x7f63f0006c00 tx=0x7f63f0055290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:09.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021f50 secure :-1 s=READY pgs=14140 cs=0 l=1 rev1=1 rx=0x7f63f0006c00 tx=0x7f63f0055290).stop 2021-03-09T10:14:09.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:09.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:09.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:09.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021f50 unknown :-1 s=CLOSED pgs=14140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:09.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:14:09.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:09.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:09.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:09.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:09.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:09.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:09.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:09.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c3a0 con 0x7f63d8003400 2021-03-09T10:14:09.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:09.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00137e0 con 0x7f63d8009130 2021-03-09T10:14:09.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:09.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017280 con 0x7f63d8005e00 2021-03-09T10:14:09.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:09.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:09.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:09.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:09.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.162+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:09.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.163+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:09.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.163+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:14:09.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.163+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:09.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.163+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:09.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.163+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:09.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.163+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:09.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.163+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:09.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:09.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:09.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:09.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:09.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:09.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:14:09.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:14:09.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:09.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:09.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cb40 con 0x7f63d8005e00 2021-03-09T10:14:09.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:09.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:09.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e980 con 0x7f63d8005e00 2021-03-09T10:14:09.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:14:09.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.164+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:09.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.165+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:14:09.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.165+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14156 cs=0 l=1 rev1=1 rx=0x7f63f0006d40 tx=0x7f63f0012350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:09.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.165+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:09.178 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:09.167+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39122 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:09.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.168+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8005e00 2021-03-09T10:14:09.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.168+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:09.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.168+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:09.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.168+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:09.180 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:09.180 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:09.180 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:09.180 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:09.181 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:09.181 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:09.181 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:09.181 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:09.181 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:09.182 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:09.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8005e00 2021-03-09T10:14:09.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.170+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:09.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.170+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:14:09.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:14:09.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:09.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:09.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:10.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:10.336+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:10.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:10.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:10.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:10.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:10.495+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:11.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:11.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:12.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:12.495+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:12.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:12.495+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:13.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:13.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:13.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:13.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:14.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:14:14.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:14.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14156 cs=0 l=1 rev1=1 rx=0x7f63f0006d40 tx=0x7f63f0012350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:14.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14156 cs=0 l=1 rev1=1 rx=0x7f63f0006d40 tx=0x7f63f0012350).stop 2021-03-09T10:14:14.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:14.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:14.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:14.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:14.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:14:14.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:14.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:14.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00686b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:14.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:14.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:14.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:14.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:14.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065eb0 con 0x7f63d8003400 2021-03-09T10:14:14.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:14.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:14:14.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:14.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63d8003840 2021-03-09T10:14:14.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:14.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:14.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:14.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:14.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.168+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:14.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.169+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00686b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:14.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.169+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:14:14.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.169+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:14.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.169+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:14.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.170+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:14.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.170+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:14.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.170+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:14.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:14.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:14.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:14.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:14.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:14.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:14:14.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:14:14.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:14.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:14.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065d20 con 0x7f63d8003840 2021-03-09T10:14:14.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:14.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:14.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e0a0 con 0x7f63d8003840 2021-03-09T10:14:14.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:14:14.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.171+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:14.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.172+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00686b0 secure :-1 s=READY pgs=14173 cs=0 l=1 rev1=1 rx=0x7f63e0017000 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:14.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.172+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:14:14.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.172+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:14.184 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:14.174+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39160 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:14.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06db40 con 0x7f63d8003840 2021-03-09T10:14:14.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.175+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:14.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.175+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:14.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.175+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:14.185 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:14.185 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:14.186 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:14.186 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:14.186 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:14.186 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:14.186 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:14.187 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:14.187 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:14.187 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:14.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc021250 con 0x7f63d8003840 2021-03-09T10:14:14.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.178+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:14.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.178+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:14:14.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06db40 con 0x7f63d8003840 2021-03-09T10:14:14.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:14.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:14.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:15.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:15.337+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:15.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:15.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:15.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:15.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:16.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:16.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:16.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:16.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:17.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:17.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:17.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:18.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:18.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:18.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:18.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:19.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00686b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:19.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00686b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:19.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00686b0 secure :-1 s=READY pgs=14173 cs=0 l=1 rev1=1 rx=0x7f63e0017000 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:19.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00686b0 secure :-1 s=READY pgs=14173 cs=0 l=1 rev1=1 rx=0x7f63e0017000 tx=0x7f63fc009a00).stop 2021-03-09T10:14:19.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:19.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:19.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00686b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:19.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00686b0 unknown :-1 s=CLOSED pgs=14173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:19.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:14:19.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:19.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:19.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:19.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:19.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:19.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:19.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:19.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d1a0 con 0x7f63d8003400 2021-03-09T10:14:19.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:19.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c3a0 con 0x7f63d8009130 2021-03-09T10:14:19.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:19.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023a90 con 0x7f63d8005e00 2021-03-09T10:14:19.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:19.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:19.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.175+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:19.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.176+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:19.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.176+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:19.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.176+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:19.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.176+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:14:19.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.176+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:19.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.176+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:19.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.176+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:19.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.176+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:19.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.176+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:19.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:19.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:19.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:19.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:19.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:19.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:14:19.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:14:19.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:19.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:19.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:14:19.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:19.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:19.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cb40 con 0x7f63d8005e00 2021-03-09T10:14:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:14:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.177+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.178+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14184 cs=0 l=1 rev1=1 rx=0x7f63f00234b0 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:19.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.178+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:14:19.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.178+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:19.191 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:19.180+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39200 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:19.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T10:14:19.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.181+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:19.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.181+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:19.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.181+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:19.193 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:19.193 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:19.193 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:19.193 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:19.194 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:19.194 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:19.194 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:19.194 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:19.195 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:19.195 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:19.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8005e00 2021-03-09T10:14:19.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.185+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:19.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.185+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:14:19.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.186+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:14:19.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:19.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:19.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:20.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:20.338+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:20.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:20.496+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:20.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:20.496+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:20.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:20.496+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:21.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:21.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:21.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:21.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:22.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:22.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:22.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:22.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:23.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:23.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:23.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:23.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14184 cs=0 l=1 rev1=1 rx=0x7f63f00234b0 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:24.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14184 cs=0 l=1 rev1=1 rx=0x7f63f00234b0 tx=0x7f63f0013b80).stop 2021-03-09T10:14:24.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:24.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:24.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:24.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:24.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:14:24.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:24.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:24.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0011b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:24.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:14:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63d8009130 2021-03-09T10:14:24.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:24.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c500 con 0x7f63d8003840 2021-03-09T10:14:24.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:24.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:24.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:24.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:24.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.181+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:24.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.182+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0011b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.182+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:14:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.182+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:24.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.182+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.182+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.182+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.182+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:24.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002ddd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:24.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:14:24.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:14:24.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:24.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:24.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e060 con 0x7f63d8003840 2021-03-09T10:14:24.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065d20 con 0x7f63d8003840 2021-03-09T10:14:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:14:24.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:24.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0011b70 secure :-1 s=READY pgs=14194 cs=0 l=1 rev1=1 rx=0x7f63e0017800 tx=0x7f63fc012eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:24.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:14:24.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.183+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:24.198 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:24.187+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39238 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:24.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0668e0 con 0x7f63d8003840 2021-03-09T10:14:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.188+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.188+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.188+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:24.199 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:24.200 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:24.200 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:24.200 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:24.201 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:24.201 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:24.201 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:24.201 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:24.202 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:24.202 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:24.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.189+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0668e0 con 0x7f63d8003840 2021-03-09T10:14:24.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.189+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:24.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.189+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:14:24.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0710f0 con 0x7f63d8003840 2021-03-09T10:14:24.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:24.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:24.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:25.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:25.339+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:25.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:25.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:25.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:25.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:26.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:26.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:26.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:26.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:27.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:27.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:27.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:27.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:28.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:28.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:28.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:28.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:29.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0011b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:29.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0011b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:29.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0011b70 secure :-1 s=READY pgs=14194 cs=0 l=1 rev1=1 rx=0x7f63e0017800 tx=0x7f63fc012eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:29.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0011b70 secure :-1 s=READY pgs=14194 cs=0 l=1 rev1=1 rx=0x7f63e0017800 tx=0x7f63fc012eb0).stop 2021-03-09T10:14:29.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:29.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:29.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0011b70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:29.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0011b70 unknown :-1 s=CLOSED pgs=14194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:29.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:14:29.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:29.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:29.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:29.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:29.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:29.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:29.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:29.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63d8003400 2021-03-09T10:14:29.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:29.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:14:29.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:29.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cf90 con 0x7f63d8005e00 2021-03-09T10:14:29.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:29.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:29.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:29.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:29.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:29.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:29.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:14:29.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:29.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:29.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:29.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:29.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.188+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:29.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:29.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:29.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:29.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:29.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:29.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:14:29.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:14:29.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:29.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:29.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e0f0 con 0x7f63d8005e00 2021-03-09T10:14:29.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:29.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:29.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:14:29.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:14:29.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.189+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:29.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.190+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14207 cs=0 l=1 rev1=1 rx=0x7f63f007b650 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:29.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.190+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:14:29.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.190+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:29.203 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:29.193+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39278 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:29.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b490 con 0x7f63d8005e00 2021-03-09T10:14:29.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.194+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:29.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.194+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:29.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.194+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:29.205 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:29.205 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:29.205 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:29.206 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:29.206 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:29.206 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:29.206 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:29.207 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:14:29.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.196+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:29.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.196+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:14:29.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:14:29.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:29.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:29.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:30.340+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:30.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:30.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:30.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:30.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:30.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:30.498+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:31.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:31.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:31.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:31.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:32.497+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:32.497+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:33.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:33.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:33.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:33.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:34.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:34.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:34.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14207 cs=0 l=1 rev1=1 rx=0x7f63f007b650 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:34.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14207 cs=0 l=1 rev1=1 rx=0x7f63f007b650 tx=0x7f63f004db20).stop 2021-03-09T10:14:34.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:34.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:34.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:34.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:34.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:14:34.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:34.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:34.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:34.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:34.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:34.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:34.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:34.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bf10 con 0x7f63d8003400 2021-03-09T10:14:34.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:34.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8009130 2021-03-09T10:14:34.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:34.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:14:34.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:34.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:34.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:34.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:34.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:34.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:34.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:14:34.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:34.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.194+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:34.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:34.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:34.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:34.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:34.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:34.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:34.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:34.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:34.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:14:34.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:14:34.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:34.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:34.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c980 con 0x7f63d8003840 2021-03-09T10:14:34.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:34.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:34.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e0f0 con 0x7f63d8003840 2021-03-09T10:14:34.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:14:34.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.195+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:34.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.196+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001d580 secure :-1 s=READY pgs=14220 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0027660).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:34.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.196+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:14:34.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.196+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:34.210 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:34.200+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39316 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:34.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.201+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:14:34.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.201+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:34.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.201+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:34.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.201+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:34.212 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:34.212 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:34.212 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:34.213 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:34.213 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:34.213 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:34.213 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:34.214 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:34.214 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:34.214 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:34.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.201+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63d8003840 2021-03-09T10:14:34.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.201+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.201+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:14:34.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0081c20 con 0x7f63d8003840 2021-03-09T10:14:34.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:34.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:34.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:35.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:35.341+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:35.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:35.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:35.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:35.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:36.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:36.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:36.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:36.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:37.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:37.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:38.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:38.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:38.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:38.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:39.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:39.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:39.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001d580 secure :-1 s=READY pgs=14220 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0027660).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:39.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001d580 secure :-1 s=READY pgs=14220 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0027660).stop 2021-03-09T10:14:39.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:39.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:39.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001d580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:39.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001d580 unknown :-1 s=CLOSED pgs=14220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:39.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:14:39.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:39.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:39.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:39.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:39.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:39.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:39.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:39.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63d8003400 2021-03-09T10:14:39.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:39.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b1e0 con 0x7f63d8009130 2021-03-09T10:14:39.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:39.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63d8005e00 2021-03-09T10:14:39.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:39.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:39.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:14:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.201+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.202+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:39.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.202+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:39.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.202+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:39.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:14:39.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:14:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029f40 con 0x7f63d8005e00 2021-03-09T10:14:39.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.203+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e060 con 0x7f63d8005e00 2021-03-09T10:14:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.204+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:14:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.204+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:39.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.204+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:14:39.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.204+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:39.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.204+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14239 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc02c220).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:39.216 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:39.206+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39356 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:39.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T10:14:39.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.206+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:39.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.206+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:39.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.206+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:39.218 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:39.218 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:39.219 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:39.219 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:39.219 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:39.219 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:39.219 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:39.220 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:39.220 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:39.220 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:39.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T10:14:39.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.208+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:39.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.208+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:14:39.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02c4d0 con 0x7f63d8005e00 2021-03-09T10:14:39.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:39.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:39.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:40.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:40.342+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:40.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:40.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:40.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:40.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:40.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:40.499+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:41.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:41.498+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:41.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:41.498+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:42.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:42.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:42.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:43.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:43.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:43.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:43.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:44.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.206+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:44.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.206+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:44.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.206+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14239 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc02c220).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:44.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.206+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14239 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc02c220).stop 2021-03-09T10:14:44.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:44.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:44.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:44.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:44.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:14:44.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:44.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:44.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:44.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:44.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:44.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:44.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:44.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:14:44.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:44.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63d8009130 2021-03-09T10:14:44.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:44.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:14:44.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:44.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:44.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:44.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:44.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.207+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:44.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:44.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:14:44.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:44.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:44.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:44.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:44.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:44.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:44.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:44.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:44.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:44.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:44.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:14:44.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:14:44.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:44.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:44.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013240 con 0x7f63d8003840 2021-03-09T10:14:44.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:44.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:44.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c980 con 0x7f63d8003840 2021-03-09T10:14:44.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:14:44.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.208+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:44.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.209+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:14:44.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.209+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:44.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 secure :-1 s=READY pgs=14247 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0055fd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:44.222 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:44.212+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39394 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:44.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:14:44.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.213+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:44.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.213+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:44.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.213+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:44.224 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:44.224 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:44.225 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:44.225 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:44.225 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:44.225 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:44.225 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:44.226 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:44.226 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:44.226 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:44.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:14:44.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.213+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:44.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.213+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:14:44.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.214+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00312f0 con 0x7f63d8003840 2021-03-09T10:14:44.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:44.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:44.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:45.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:45.343+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:45.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:45.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:45.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:45.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:46.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:46.499+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:46.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:46.499+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:47.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:47.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:47.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:47.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:48.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:48.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:48.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:48.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:49.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:49.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:49.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 secure :-1 s=READY pgs=14247 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0055fd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:49.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 secure :-1 s=READY pgs=14247 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0055fd0).stop 2021-03-09T10:14:49.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:49.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:49.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:49.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 unknown :-1 s=CLOSED pgs=14247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:49.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:14:49.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:49.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:49.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:49.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:49.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:49.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:49.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:49.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63d8003400 2021-03-09T10:14:49.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:49.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c810 con 0x7f63d8009130 2021-03-09T10:14:49.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:49.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023770 con 0x7f63d8005e00 2021-03-09T10:14:49.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:49.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:49.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.213+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:49.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.214+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:49.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.214+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:49.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.214+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:49.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.214+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:14:49.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.214+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:49.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.214+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:49.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.214+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:49.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.214+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:49.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.214+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:49.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:49.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:49.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:49.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:49.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:49.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:14:49.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:14:49.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:49.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:49.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:14:49.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:49.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:49.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013240 con 0x7f63d8005e00 2021-03-09T10:14:49.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:14:49.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.215+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:49.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14265 cs=0 l=1 rev1=1 rx=0x7f63f0052010 tx=0x7f63f007e510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:49.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.216+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:14:49.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.216+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:49.230 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:49.219+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39433 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:49.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.219+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007c560 con 0x7f63d8005e00 2021-03-09T10:14:49.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.219+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:49.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.219+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:49.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.219+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:49.231 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:49.232 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:49.232 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:49.232 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:49.232 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:49.233 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:49.233 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:49.233 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:49.233 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:49.233 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:49.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8005e00 2021-03-09T10:14:49.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.222+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:49.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.222+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:14:49.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T10:14:49.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:49.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:49.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:50.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:50.344+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:50.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:50.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:50.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:50.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:50.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:50.500+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:51.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:51.500+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:51.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:51.500+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:52.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:52.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:52.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:52.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:53.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:53.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:53.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:53.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:54.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:14:54.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:54.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14265 cs=0 l=1 rev1=1 rx=0x7f63f0052010 tx=0x7f63f007e510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:54.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14265 cs=0 l=1 rev1=1 rx=0x7f63f0052010 tx=0x7f63f007e510).stop 2021-03-09T10:14:54.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:54.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:54.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:54.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:54.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:14:54.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:54.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:54.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:54.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:54.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:54.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:54.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:54.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bdc0 con 0x7f63d8003400 2021-03-09T10:14:54.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:54.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:14:54.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:54.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d280 con 0x7f63d8003840 2021-03-09T10:14:54.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:54.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:54.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:54.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:54.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.220+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:54.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:54.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:14:54.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:54.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:54.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:54.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:54.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:54.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:54.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:54.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:54.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:54.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:54.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:14:54.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:14:54.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:54.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:54.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c980 con 0x7f63d8003840 2021-03-09T10:14:54.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:54.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:54.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:14:54.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:14:54.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.222+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:54.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.223+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001740 secure :-1 s=READY pgs=14281 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f00284b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:54.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.223+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:14:54.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.223+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:54.237 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:54.224+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39472 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:54.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.227+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040640 con 0x7f63d8003840 2021-03-09T10:14:54.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.227+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:54.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.227+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:54.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.227+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:54.239 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:54.239 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:54.239 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:54.239 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:54.240 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:54.240 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:54.240 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:54.240 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:54.240 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:54.241 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:54.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.227+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8003840 2021-03-09T10:14:54.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.227+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:54.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.227+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:14:54.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.228+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:14:54.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:54.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:54.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:55.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:55.345+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:14:55.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:55.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:55.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:55.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:56.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:56.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:56.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:56.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:57.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:57.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:57.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:57.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:58.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:58.501+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:58.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:58.501+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:59.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:14:59.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:14:59.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001740 secure :-1 s=READY pgs=14281 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f00284b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:14:59.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001740 secure :-1 s=READY pgs=14281 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f00284b0).stop 2021-03-09T10:14:59.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:59.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:14:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:14:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001740 unknown :-1 s=CLOSED pgs=14281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:14:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:14:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:59.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:14:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:14:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:14:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:14:59.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021c20 con 0x7f63d8003400 2021-03-09T10:14:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:59.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00147c0 con 0x7f63d8009130 2021-03-09T10:14:59.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:14:59.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b960 con 0x7f63d8005e00 2021-03-09T10:14:59.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:14:59.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:14:59.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:59.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:14:59.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:14:59.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:14:59.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:14:59.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:14:59.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:14:59.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:14:59.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:14:59.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.226+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:14:59.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:14:59.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:59.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:59.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:14:59.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:14:59.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:14:59.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:14:59.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:14:59.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:59.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e0f0 con 0x7f63d8005e00 2021-03-09T10:14:59.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:14:59.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:14:59.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c980 con 0x7f63d8005e00 2021-03-09T10:14:59.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:14:59.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:14:59.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14289 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f002fd40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:14:59.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:14:59.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.227+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:14:59.242 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:14:59.231+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39510 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:14:59.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:14:59.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.231+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:14:59.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.231+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:14:59.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.231+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:14:59.244 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:14:59.244 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:14:59.245 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:59.245 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:14:59.245 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:14:59.245 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:14:59.245 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:14:59.246 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:14:59.246 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:14:59.246 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:14:59.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:14:59.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.231+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:14:59.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.231+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:14:59.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8005e00 2021-03-09T10:14:59.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:14:59.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:14:59.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:00.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:00.346+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:00.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:00.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:00.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:00.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:00.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:00.502+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:01.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:01.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:01.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:01.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:02.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:02.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:02.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:02.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:03.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:03.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:03.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:03.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:04.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:04.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:04.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14289 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f002fd40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:04.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14289 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f002fd40).stop 2021-03-09T10:15:04.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:04.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:04.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:04.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:04.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:15:04.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:04.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:04.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.232+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:04.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:04.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:04.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:04.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:04.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:15:04.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:04.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63d8009130 2021-03-09T10:15:04.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:04.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003840 2021-03-09T10:15:04.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:04.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:04.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:04.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:04.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.233+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:04.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.234+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:04.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.234+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:15:04.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.234+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:04.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.234+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:04.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.234+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:04.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.234+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:04.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.234+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:04.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:04.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:04.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:04.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:04.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:04.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:15:04.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:15:04.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:04.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:04.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:15:04.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:04.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:04.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029f40 con 0x7f63d8003840 2021-03-09T10:15:04.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:15:04.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.235+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:04.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022ab0 secure :-1 s=READY pgs=14306 cs=0 l=1 rev1=1 rx=0x7f63e000d4e0 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:04.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.236+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:15:04.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.236+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:04.249 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:04.238+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39550 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:04.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.238+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63d8003840 2021-03-09T10:15:04.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.238+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:04.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.238+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:04.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.238+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:04.251 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:04.251 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:04.251 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:04.251 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:04.252 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:04.252 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:04.252 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:04.252 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:04.253 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:04.253 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:04.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.242+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:15:04.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.242+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:04.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.242+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:15:04.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f460 con 0x7f63d8003840 2021-03-09T10:15:04.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:04.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:04.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:05.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:05.347+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:05.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:05.502+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:05.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:05.502+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:06.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:06.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:06.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:06.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:07.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:07.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:07.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:07.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:08.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:08.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:08.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:08.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:09.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:09.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:09.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022ab0 secure :-1 s=READY pgs=14306 cs=0 l=1 rev1=1 rx=0x7f63e000d4e0 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:09.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022ab0 secure :-1 s=READY pgs=14306 cs=0 l=1 rev1=1 rx=0x7f63e000d4e0 tx=0x7f63fc009a00).stop 2021-03-09T10:15:09.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:09.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:09.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:09.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022ab0 unknown :-1 s=CLOSED pgs=14306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:09.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:15:09.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:09.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:09.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:09.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:09.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:09.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:09.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:09.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.238+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a170 con 0x7f63d8003400 2021-03-09T10:15:09.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:09.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:15:09.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:09.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e970 con 0x7f63d8005e00 2021-03-09T10:15:09.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:09.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:09.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:09.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:09.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:09.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:09.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:15:09.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:09.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.239+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:09.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:09.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:09.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:09.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:09.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:09.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:15:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:15:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010cb0 con 0x7f63d8005e00 2021-03-09T10:15:09.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:09.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:09.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e0f0 con 0x7f63d8005e00 2021-03-09T10:15:09.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:15:09.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.240+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:09.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.242+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:15:09.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.242+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:09.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.242+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14320 cs=0 l=1 rev1=1 rx=0x7f63f005f9c0 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:09.253 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:09.244+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39588 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:09.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00116b0 con 0x7f63d8005e00 2021-03-09T10:15:09.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.245+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:09.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.245+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:09.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.245+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:09.255 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:09.255 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:09.255 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:09.256 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:09.256 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:09.256 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:09.256 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:09.257 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:09.257 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:09.257 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:09.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.247+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00116b0 con 0x7f63d8005e00 2021-03-09T10:15:09.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.248+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:09.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.248+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:15:09.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:15:09.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:09.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:09.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:10.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:10.348+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:10.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:10.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:10.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:10.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:10.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:10.503+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:11.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:11.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:11.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:11.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:12.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:12.503+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:12.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:12.503+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:13.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:13.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:13.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:13.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:14.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.244+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:14.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.244+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:14.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.244+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14320 cs=0 l=1 rev1=1 rx=0x7f63f005f9c0 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:14.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.244+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14320 cs=0 l=1 rev1=1 rx=0x7f63f005f9c0 tx=0x7f63f004d180).stop 2021-03-09T10:15:14.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:14.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:14.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:14.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:14.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:15:14.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:14.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:14.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:14.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:14.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:14.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:14.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:14.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b960 con 0x7f63d8003400 2021-03-09T10:15:14.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:14.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bdc0 con 0x7f63d8009130 2021-03-09T10:15:14.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:14.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003840 2021-03-09T10:15:14.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:14.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:14.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:14.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:14.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:14.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:14.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:15:14.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:14.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.245+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:14.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:14.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:14.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:14.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:14.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:14.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:14.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:14.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:14.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:15:14.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:15:14.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:14.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:14.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e0f0 con 0x7f63d8003840 2021-03-09T10:15:14.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.247+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:14.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.247+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:14.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.247+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:15:14.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.247+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:15:14.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.247+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:14.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.247+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:15:14.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.247+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012d70 secure :-1 s=READY pgs=14331 cs=0 l=1 rev1=1 rx=0x7f63e0011ab0 tx=0x7f63fc013c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:14.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.247+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:14.259 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:14.250+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39628 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:14.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013e60 con 0x7f63d8003840 2021-03-09T10:15:14.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.250+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:14.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.250+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:14.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.250+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:14.261 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:14.261 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:14.261 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:14.262 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:14.262 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:14.262 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:14.263 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:14.263 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:14.263 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:14.263 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:14.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013e60 con 0x7f63d8003840 2021-03-09T10:15:14.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.250+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:14.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.251+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:15:14.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.252+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc05d290 con 0x7f63d8003840 2021-03-09T10:15:14.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:14.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:14.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:15.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:15.349+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:15.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:15.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:15.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:15.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:16.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:16.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:16.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:16.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:17.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:17.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:17.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:17.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:18.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:18.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:18.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:18.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:19.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:19.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:19.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012d70 secure :-1 s=READY pgs=14331 cs=0 l=1 rev1=1 rx=0x7f63e0011ab0 tx=0x7f63fc013c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:19.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012d70 secure :-1 s=READY pgs=14331 cs=0 l=1 rev1=1 rx=0x7f63e0011ab0 tx=0x7f63fc013c60).stop 2021-03-09T10:15:19.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:19.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:19.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012d70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:19.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012d70 unknown :-1 s=CLOSED pgs=14331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:19.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:15:19.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:19.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:19.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:19.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:19.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:19.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:19.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:19.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003400 2021-03-09T10:15:19.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:19.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a5a0 con 0x7f63d8009130 2021-03-09T10:15:19.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:19.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:15:19.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:19.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:19.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:19.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:19.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:19.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:19.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:15:19.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:19.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.251+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:19.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.252+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:19.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.252+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:19.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.252+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:19.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.252+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:19.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:19.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:19.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:19.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:19.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:15:19.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:15:19.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:19.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:19.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016aa0 con 0x7f63d8005e00 2021-03-09T10:15:19.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:19.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:19.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e0f0 con 0x7f63d8005e00 2021-03-09T10:15:19.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:15:19.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:19.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14345 cs=0 l=1 rev1=1 rx=0x7f63fc0099d0 tx=0x7f63fc02cf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:19.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:15:19.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.253+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:19.277 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:19.256+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39666 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:19.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.258+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02cac0 con 0x7f63d8005e00 2021-03-09T10:15:19.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.258+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:19.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.258+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:19.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.258+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:19.280 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:19.280 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:19.281 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:19.281 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:19.282 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:19.282 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:19.282 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:19.283 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:19.283 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:19.284 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:19.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.260+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02cac0 con 0x7f63d8005e00 2021-03-09T10:15:19.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.260+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:19.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.260+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:15:19.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8005e00 2021-03-09T10:15:19.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:19.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:19.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:20.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:20.350+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:20.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:20.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:20.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:20.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:20.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:20.504+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:21.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:21.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:21.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:21.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:22.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:22.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:22.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:23.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:23.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:23.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:23.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:24.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.256+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.256+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.256+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14345 cs=0 l=1 rev1=1 rx=0x7f63fc0099d0 tx=0x7f63fc02cf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 secure :-1 s=READY pgs=14345 cs=0 l=1 rev1=1 rx=0x7f63fc0099d0 tx=0x7f63fc02cf90).stop 2021-03-09T10:15:24.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0064240 unknown :-1 s=CLOSED pgs=14345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:24.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:15:24.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:24.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:24.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:24.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:24.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:24.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:24.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:24.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012730 con 0x7f63d8003400 2021-03-09T10:15:24.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:24.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:15:24.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:24.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bdc0 con 0x7f63d8003840 2021-03-09T10:15:24.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:24.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:24.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:24.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:24.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.257+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:24.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.258+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.258+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:15:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.258+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.258+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:24.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.258+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.258+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.258+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:24.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e001ada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:15:24.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:15:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:15:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:24.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010cb0 con 0x7f63d8003840 2021-03-09T10:15:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:15:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.259+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:24.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.260+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:15:24.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.260+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:24.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.260+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 secure :-1 s=READY pgs=14364 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0009510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:24.271 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:24.263+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39706 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:24.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002fdb0 con 0x7f63d8003840 2021-03-09T10:15:24.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.263+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:24.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.263+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:24.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.263+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:24.273 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:24.273 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:24.273 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:24.273 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:24.274 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:24.274 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:24.274 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:24.274 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:24.275 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:24.275 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:24.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f003b480 con 0x7f63d8003840 2021-03-09T10:15:24.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.267+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:24.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.267+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:15:24.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.268+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:15:24.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:24.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:24.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:25.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:25.351+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:25.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:25.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:25.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:25.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:26.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:26.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:27.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:27.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:27.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:27.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:28.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:28.504+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:28.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:28.504+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:29.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:29.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:29.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 secure :-1 s=READY pgs=14364 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0009510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:29.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 secure :-1 s=READY pgs=14364 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0009510).stop 2021-03-09T10:15:29.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:29.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:29.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c0f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:29.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c0f0 unknown :-1 s=CLOSED pgs=14364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:29.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:15:29.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:29.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:29.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:29.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:29.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:29.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:29.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:29.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015f80 con 0x7f63d8003400 2021-03-09T10:15:29.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:29.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00174c0 con 0x7f63d8009130 2021-03-09T10:15:29.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:29.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:15:29.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:29.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:29.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.262+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:29.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.263+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:29.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.263+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:29.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.263+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:29.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.263+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:15:29.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.263+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:29.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.263+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:29.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.264+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:29.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.264+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:29.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.264+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:29.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:29.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:29.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:29.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:29.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:29.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:15:29.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:15:29.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:29.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:29.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bc30 con 0x7f63d8005e00 2021-03-09T10:15:29.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:29.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:29.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:15:29.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:15:29.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:29.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:15:29.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:29.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.265+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001ada0 secure :-1 s=READY pgs=14379 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f004cbf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:29.277 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:29.268+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39744 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:29.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002fbe0 con 0x7f63d8005e00 2021-03-09T10:15:29.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.269+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:29.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.269+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:29.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.269+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:29.279 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:29.279 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:29.279 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:29.279 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:29.280 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:29.280 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:29.280 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:29.280 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:29.280 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:29.281 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:29.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8005e00 2021-03-09T10:15:29.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.269+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:29.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.269+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:15:29.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8005e00 2021-03-09T10:15:29.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:29.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:29.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:30.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:30.351+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:30.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:30.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:30.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:30.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:30.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:30.505+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:31.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:31.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:31.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:31.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:32.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:32.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:32.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:32.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:33.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:33.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:33.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:33.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:34.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:34.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:34.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001ada0 secure :-1 s=READY pgs=14379 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f004cbf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:34.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001ada0 secure :-1 s=READY pgs=14379 cs=0 l=1 rev1=1 rx=0x7f63f005eff0 tx=0x7f63f004cbf0).stop 2021-03-09T10:15:34.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:34.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:34.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001ada0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:34.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001ada0 unknown :-1 s=CLOSED pgs=14379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:34.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:15:34.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:34.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:34.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:34.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:34.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:34.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:34.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:34.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d040 con 0x7f63d8003400 2021-03-09T10:15:34.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:34.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007ae50 con 0x7f63d8009130 2021-03-09T10:15:34.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:34.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00137e0 con 0x7f63d8003840 2021-03-09T10:15:34.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:34.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:34.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:34.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:34.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.269+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:34.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.270+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:34.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.270+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:15:34.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.270+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:34.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.270+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:34.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:34.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:34.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:34.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.271+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:34.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:34.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:34.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:34.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:34.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:15:34.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:15:34.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:34.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:34.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:15:34.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:34.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:34.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bc30 con 0x7f63d8003840 2021-03-09T10:15:34.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:15:34.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.272+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:34.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.273+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 secure :-1 s=READY pgs=14393 cs=0 l=1 rev1=1 rx=0x7f63f0000c90 tx=0x7f63f00066e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:34.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.273+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:15:34.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.273+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:34.284 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:34.275+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39784 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:34.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.276+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:15:34.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.276+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:34.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.276+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:34.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.276+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:34.286 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:34.287 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:34.287 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:34.287 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:34.287 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:34.288 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:34.288 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:34.288 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:34.289 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:34.289 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:34.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8003840 2021-03-09T10:15:34.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.277+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:34.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.277+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:15:34.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.278+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:15:34.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:34.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:34.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:35.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:35.352+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:35.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:35.505+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:35.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:35.505+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:36.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:36.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:36.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:36.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:37.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:37.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:38.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:38.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:38.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:38.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:39.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.276+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:39.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.276+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:39.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.276+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 secure :-1 s=READY pgs=14393 cs=0 l=1 rev1=1 rx=0x7f63f0000c90 tx=0x7f63f00066e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:39.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.276+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 secure :-1 s=READY pgs=14393 cs=0 l=1 rev1=1 rx=0x7f63f0000c90 tx=0x7f63f00066e0).stop 2021-03-09T10:15:39.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:39.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:39.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:39.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 unknown :-1 s=CLOSED pgs=14393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:39.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:15:39.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:39.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:39.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:39.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:39.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:39.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:39.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:39.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00252f0 con 0x7f63d8003400 2021-03-09T10:15:39.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:39.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c4f0 con 0x7f63d8009130 2021-03-09T10:15:39.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:39.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00151c0 con 0x7f63d8005e00 2021-03-09T10:15:39.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:39.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:39.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:39.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:39.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.277+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:39.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.278+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:39.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.278+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:15:39.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.278+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:39.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.278+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:39.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:39.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:39.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:39.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:39.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:39.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:39.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:39.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:39.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:15:39.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:15:39.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:39.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:39.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001cc50 con 0x7f63d8005e00 2021-03-09T10:15:39.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:39.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:39.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:15:39.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:15:39.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:39.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14408 cs=0 l=1 rev1=1 rx=0x7f63f0053040 tx=0x7f63f004dab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:39.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.280+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:15:39.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.280+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:39.293 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:39.282+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39822 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:39.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8005e00 2021-03-09T10:15:39.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.282+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:39.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.282+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:39.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.282+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:39.295 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:39.295 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:39.295 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:39.296 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:39.296 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:39.296 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:39.296 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:39.297 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:39.297 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:39.297 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:39.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8005e00 2021-03-09T10:15:39.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.286+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:39.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.286+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:15:39.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005fd70 con 0x7f63d8005e00 2021-03-09T10:15:39.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:39.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:39.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:40.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:40.353+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:40.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:40.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:40.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:40.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:40.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:40.506+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:41.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:41.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:41.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:41.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:42.506+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:42.506+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:43.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:43.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:43.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:43.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.282+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:44.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14408 cs=0 l=1 rev1=1 rx=0x7f63f0053040 tx=0x7f63f004dab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14408 cs=0 l=1 rev1=1 rx=0x7f63f0053040 tx=0x7f63f004dab0).stop 2021-03-09T10:15:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:44.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:15:44.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:44.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:44.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:44.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:44.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:44.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:44.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:44.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00174c0 con 0x7f63d8003400 2021-03-09T10:15:44.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:44.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006def0 con 0x7f63d8009130 2021-03-09T10:15:44.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:44.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007ae50 con 0x7f63d8003840 2021-03-09T10:15:44.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:44.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:44.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:44.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:44.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.283+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:44.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:44.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:15:44.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:44.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:44.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:44.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:44.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:44.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:44.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:44.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:44.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:44.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:44.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:15:44.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:15:44.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:44.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:44.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0021bd0 con 0x7f63d8003840 2021-03-09T10:15:44.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:44.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:44.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001cc50 con 0x7f63d8003840 2021-03-09T10:15:44.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:15:44.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.284+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:44.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.285+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:15:44.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.285+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:44.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.285+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 secure :-1 s=READY pgs=14414 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f00563b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:44.298 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:44.288+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39862 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:44.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.289+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028e50 con 0x7f63d8003840 2021-03-09T10:15:44.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.289+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:44.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.289+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:44.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.289+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:44.302 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:44.302 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:44.302 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:44.303 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:44.303 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:44.303 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:44.304 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:44.304 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:44.304 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:44.305 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:44.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.291+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030240 con 0x7f63d8003840 2021-03-09T10:15:44.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.291+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:44.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.291+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:15:44.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0031da0 con 0x7f63d8003840 2021-03-09T10:15:44.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:44.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:44.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:45.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:45.354+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:45.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:45.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:45.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:45.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:46.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:46.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:46.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:46.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:47.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:47.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:47.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:47.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:48.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:48.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:48.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:48.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:49.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:49.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:49.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 secure :-1 s=READY pgs=14414 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f00563b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:49.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 secure :-1 s=READY pgs=14414 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f00563b0).stop 2021-03-09T10:15:49.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:49.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:49.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:49.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 unknown :-1 s=CLOSED pgs=14414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:49.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:15:49.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:49.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:49.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:49.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:49.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:49.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:49.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:49.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012730 con 0x7f63d8003400 2021-03-09T10:15:49.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:49.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00151c0 con 0x7f63d8009130 2021-03-09T10:15:49.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:49.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00252f0 con 0x7f63d8005e00 2021-03-09T10:15:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:49.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.289+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:15:49.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:49.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:49.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:49.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:49.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:49.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:15:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:15:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:49.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:15:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:49.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.290+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0021bd0 con 0x7f63d8005e00 2021-03-09T10:15:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.291+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:15:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.291+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:49.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.291+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:15:49.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.291+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:49.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.291+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14429 cs=0 l=1 rev1=1 rx=0x7f63f000c010 tx=0x7f63f00095f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:49.305 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:49.294+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39900 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:49.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:15:49.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.295+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:49.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.295+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:49.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.295+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:49.307 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:49.308 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:49.308 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:49.308 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:49.308 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:49.308 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:49.309 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:49.309 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:49.309 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:49.309 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:49.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:15:49.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.301+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:49.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.301+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:15:49.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:15:49.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:49.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:49.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:50.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:50.355+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:50.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:50.507+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:50.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:50.507+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:50.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:50.507+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:51.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:51.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:51.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:51.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:52.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:52.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:52.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:52.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:53.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:53.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:53.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:53.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:54.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:15:54.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:54.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14429 cs=0 l=1 rev1=1 rx=0x7f63f000c010 tx=0x7f63f00095f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:54.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14429 cs=0 l=1 rev1=1 rx=0x7f63f000c010 tx=0x7f63f00095f0).stop 2021-03-09T10:15:54.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:54.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:54.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:54.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:54.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:15:54.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:54.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:54.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:54.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:54.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:54.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:54.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:54.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00181c0 con 0x7f63d8003400 2021-03-09T10:15:54.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:54.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:15:54.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:54.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00151c0 con 0x7f63d8003840 2021-03-09T10:15:54.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:54.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:54.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:54.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:54.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.295+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:54.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:54.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:15:54.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:54.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:54.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:54.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:54.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:54.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:54.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:54.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:54.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:54.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:54.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:15:54.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:15:54.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:54.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:54.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63d8003840 2021-03-09T10:15:54.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:54.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:54.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.296+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016aa0 con 0x7f63d8003840 2021-03-09T10:15:54.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.297+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:15:54.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.297+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:54.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.297+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:15:54.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.297+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:54.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.297+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 secure :-1 s=READY pgs=14442 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:54.309 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:54.300+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39940 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:54.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013de0 con 0x7f63d8003840 2021-03-09T10:15:54.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.301+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:54.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.301+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:54.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.301+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:54.311 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:54.311 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:54.312 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:54.312 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:54.312 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:54.312 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:54.312 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:54.313 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:54.313 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:54.313 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:54.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02ccf0 con 0x7f63d8003840 2021-03-09T10:15:54.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.301+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:54.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.301+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:15:54.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.302+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013de0 con 0x7f63d8003840 2021-03-09T10:15:54.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:54.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:54.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:55.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:55.356+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:15:55.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:55.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:55.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:55.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:56.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:56.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:56.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:56.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:57.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:57.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:57.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:57.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:58.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:58.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:58.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:58.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:59.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:15:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:15:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 secure :-1 s=READY pgs=14442 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:15:59.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 secure :-1 s=READY pgs=14442 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc003a50).stop 2021-03-09T10:15:59.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:59.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:15:59.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:15:59.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c800 unknown :-1 s=CLOSED pgs=14442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:15:59.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:15:59.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:59.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:15:59.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:59.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:15:59.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:15:59.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:15:59.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:59.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8003400 2021-03-09T10:15:59.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:59.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00137e0 con 0x7f63d8009130 2021-03-09T10:15:59.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:15:59.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:15:59.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:15:59.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:15:59.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:59.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:15:59.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.301+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:15:59.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.302+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:15:59.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.302+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:15:59.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.302+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:15:59.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.302+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:15:59.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:15:59.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:15:59.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:15:59.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:15:59.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:59.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:59.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:15:59.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:15:59.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:15:59.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:15:59.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:15:59.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:59.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079460 con 0x7f63d8005e00 2021-03-09T10:15:59.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:15:59.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:15:59.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:15:59.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:15:59.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.303+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:15:59.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.304+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14459 cs=0 l=1 rev1=1 rx=0x7f63f00035e0 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:15:59.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.304+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:15:59.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.304+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:15:59.317 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:15:59.306+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 39978 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:15:59.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:15:59.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.307+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:15:59.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.307+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:15:59.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.307+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:15:59.319 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:15:59.319 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:15:59.319 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:59.320 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:15:59.320 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:15:59.320 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:15:59.320 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:15:59.321 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:15:59.321 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:15:59.321 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:15:59.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:15:59.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.310+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:15:59.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.310+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:15:59.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.312+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007bd80 con 0x7f63d8005e00 2021-03-09T10:15:59.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.508+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:15:59.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:15:59.508+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:00.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:00.357+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:00.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:00.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:00.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:00.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:00.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:00.509+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:01.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:01.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:01.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:01.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:02.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:02.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:02.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:02.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:03.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:03.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:03.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:03.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:04.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:04.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:04.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14459 cs=0 l=1 rev1=1 rx=0x7f63f00035e0 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:04.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14459 cs=0 l=1 rev1=1 rx=0x7f63f00035e0 tx=0x7f63f0013b80).stop 2021-03-09T10:16:04.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:04.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:04.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:04.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:04.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:16:04.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:04.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:04.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:04.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:04.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:04.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:04.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:04.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:16:04.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:04.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068330 con 0x7f63d8009130 2021-03-09T10:16:04.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:04.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00252f0 con 0x7f63d8003840 2021-03-09T10:16:04.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:04.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:04.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.307+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:04.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.308+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:04.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.308+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:04.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.308+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:04.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.308+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:16:04.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.308+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:04.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.308+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:04.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.308+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:04.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.308+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:04.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.308+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:04.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:04.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:04.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:04.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:04.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:04.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:04.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:16:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079460 con 0x7f63d8003840 2021-03-09T10:16:04.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:04.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:04.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.309+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b790 secure :-1 s=READY pgs=14470 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:04.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.310+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:16:04.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.310+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:04.323 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:04.313+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40018 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:04.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:16:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.313+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.313+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.314+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:04.324 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:04.325 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:04.326 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:04.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:16:04.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.316+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:04.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.316+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:16:04.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.318+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:16:04.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:04.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:04.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:05.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:05.358+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:05.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:05.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:05.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:05.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:06.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:06.509+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:06.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:06.509+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:07.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:07.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:07.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:07.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:08.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:08.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:08.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:08.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:09.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:09.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:09.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b790 secure :-1 s=READY pgs=14470 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:09.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b790 secure :-1 s=READY pgs=14470 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004db20).stop 2021-03-09T10:16:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:09.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:09.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b790 unknown :-1 s=CLOSED pgs=14470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:09.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:16:09.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:09.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:09.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:09.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:09.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:09.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:09.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:09.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:16:09.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:09.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bdf0 con 0x7f63d8009130 2021-03-09T10:16:09.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:09.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:16:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.313+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.314+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.314+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.314+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:09.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.314+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:09.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.314+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:09.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:16:09.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:09.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:09.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:09.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:09.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:09.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:09.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:09.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:09.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:09.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:09.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:09.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:09.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:09.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:09.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015170 con 0x7f63d8005e00 2021-03-09T10:16:09.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:09.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:09.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:16:09.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:09.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.315+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:09.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14489 cs=0 l=1 rev1=1 rx=0x7f63f00035e0 tx=0x7f63f0041340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:09.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.317+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:16:09.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.317+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:09.332 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:09.320+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40056 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:09.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:16:09.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.321+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:09.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.321+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:09.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.321+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:09.334 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:09.334 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:09.334 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:09.335 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:09.336 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:09.336 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:09.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:16:09.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.324+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:09.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.324+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:16:09.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007bd80 con 0x7f63d8005e00 2021-03-09T10:16:09.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:09.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:09.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:10.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:10.359+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:10.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:10.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:10.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:10.510+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:11.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:11.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:11.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:11.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:12.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:12.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:12.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:12.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:13.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:13.510+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:13.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:13.510+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:14.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:14.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:14.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14489 cs=0 l=1 rev1=1 rx=0x7f63f00035e0 tx=0x7f63f0041340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:14.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14489 cs=0 l=1 rev1=1 rx=0x7f63f00035e0 tx=0x7f63f0041340).stop 2021-03-09T10:16:14.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:14.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:14.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:14.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:14.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:16:14.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:14.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:14.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:14.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:14.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:14.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:14.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:14.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023970 con 0x7f63d8003400 2021-03-09T10:16:14.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:14.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00252f0 con 0x7f63d8009130 2021-03-09T10:16:14.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:14.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068330 con 0x7f63d8003840 2021-03-09T10:16:14.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:14.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:14.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:14.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.320+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:14.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.321+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:14.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.321+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:14.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.321+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:16:14.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.321+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:14.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.321+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:14.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.321+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:14.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.321+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:14.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.321+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:14.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:14.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:14.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:14.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:14.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:14.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:14.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:14.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:14.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:14.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018010 con 0x7f63d8003840 2021-03-09T10:16:14.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:14.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:14.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015170 con 0x7f63d8003840 2021-03-09T10:16:14.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:14.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:14.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:16:14.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:14.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.322+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a050 secure :-1 s=READY pgs=14497 cs=0 l=1 rev1=1 rx=0x7f63f007c4f0 tx=0x7f63f0006ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:14.334 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:14.326+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40096 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:14.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8003840 2021-03-09T10:16:14.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.327+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:14.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.327+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:14.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.327+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:14.335 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:14.336 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:14.336 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:14.336 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:14.336 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:14.336 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:14.337 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:14.337 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:14.337 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:14.337 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:14.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f001d330 con 0x7f63d8003840 2021-03-09T10:16:14.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.327+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:14.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.327+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:16:14.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.329+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8003840 2021-03-09T10:16:14.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:14.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:14.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:15.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:15.360+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:15.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:15.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:15.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:15.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:16.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:16.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:16.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:16.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:17.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:17.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:17.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:17.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:18.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:18.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:18.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:18.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:19.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:16:19.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:19.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a050 secure :-1 s=READY pgs=14497 cs=0 l=1 rev1=1 rx=0x7f63f007c4f0 tx=0x7f63f0006ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:19.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a050 secure :-1 s=READY pgs=14497 cs=0 l=1 rev1=1 rx=0x7f63f007c4f0 tx=0x7f63f0006ca0).stop 2021-03-09T10:16:19.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:19.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:19.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:19.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a050 unknown :-1 s=CLOSED pgs=14497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:19.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:16:19.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:19.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:19.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:19.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:19.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:19.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:19.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:19.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011770 con 0x7f63d8003400 2021-03-09T10:16:19.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:19.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:16:19.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:19.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00098e0 con 0x7f63d8005e00 2021-03-09T10:16:19.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.326+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:19.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:19.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:19.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:19.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:19.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:19.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:16:19.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:19.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:19.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.327+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:19.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:19.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:16:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:19.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:19.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018010 con 0x7f63d8005e00 2021-03-09T10:16:19.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:19.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:19.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14509 cs=0 l=1 rev1=1 rx=0x7f63e000d810 tx=0x7f63f007ff10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:19.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:16:19.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.328+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:19.341 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:19.332+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40134 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:19.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.332+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00884c0 con 0x7f63d8005e00 2021-03-09T10:16:19.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.332+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:19.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.332+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.332+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:19.343 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:19.344 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:19.345 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.332+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T10:16:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.332+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:19.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.332+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:16:19.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f63d8005e00 2021-03-09T10:16:19.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:19.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:19.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:20.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:20.361+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:20.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:20.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:20.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:20.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:20.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:20.511+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:21.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:21.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:21.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:21.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:22.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:22.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:22.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:22.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:23.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:23.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:23.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:23.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:24.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:24.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:24.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14509 cs=0 l=1 rev1=1 rx=0x7f63e000d810 tx=0x7f63f007ff10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:24.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14509 cs=0 l=1 rev1=1 rx=0x7f63e000d810 tx=0x7f63f007ff10).stop 2021-03-09T10:16:24.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:24.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:24.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:24.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:24.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:16:24.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:24.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:24.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:24.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:24.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:24.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:24.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:24.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:16:24.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:24.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021e20 con 0x7f63d8009130 2021-03-09T10:16:24.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:24.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068330 con 0x7f63d8003840 2021-03-09T10:16:24.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:24.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:24.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:24.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:24.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.333+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:24.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:24.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:16:24.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:24.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:24.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:24.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:24.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:24.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:24.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:24.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:24.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:24.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:24.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:24.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:24.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:24.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:24.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b960 con 0x7f63d8003840 2021-03-09T10:16:24.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:24.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:24.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:16:24.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:24.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.334+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:24.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.336+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002fd90 secure :-1 s=READY pgs=14526 cs=0 l=1 rev1=1 rx=0x7f63f005f8b0 tx=0x7f63f0041590).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:24.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.336+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:16:24.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.336+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:24.349 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:24.339+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40174 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:24.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:16:24.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.341+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:24.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.341+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:24.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.341+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:24.351 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:24.351 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:24.351 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:24.351 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:24.352 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:24.352 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:24.352 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:24.352 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:24.352 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:24.353 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:24.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a430 con 0x7f63d8003840 2021-03-09T10:16:24.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.344+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:24.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.344+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:16:24.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:16:24.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.511+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:24.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:24.511+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:25.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:25.362+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:25.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:25.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:25.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:25.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:26.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:26.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:26.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:26.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:27.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:27.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:27.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:27.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:28.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:28.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:28.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:28.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:29.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:29.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:29.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002fd90 secure :-1 s=READY pgs=14526 cs=0 l=1 rev1=1 rx=0x7f63f005f8b0 tx=0x7f63f0041590).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:29.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002fd90 secure :-1 s=READY pgs=14526 cs=0 l=1 rev1=1 rx=0x7f63f005f8b0 tx=0x7f63f0041590).stop 2021-03-09T10:16:29.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:29.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002fd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002fd90 unknown :-1 s=CLOSED pgs=14526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:16:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00196b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:29.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:29.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:29.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:29.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:29.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:29.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:29.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00252f0 con 0x7f63d8003400 2021-03-09T10:16:29.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:29.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:16:29.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:29.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025170 con 0x7f63d8005e00 2021-03-09T10:16:29.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:29.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00196b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:16:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:29.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.340+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00296a0 con 0x7f63d8005e00 2021-03-09T10:16:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b960 con 0x7f63d8005e00 2021-03-09T10:16:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:16:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.341+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00196b0 secure :-1 s=READY pgs=14537 cs=0 l=1 rev1=1 rx=0x7f63e001d4e0 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:29.355 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:29.345+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40212 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0012560 con 0x7f63d8005e00 2021-03-09T10:16:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.345+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.345+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.345+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:29.357 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:29.357 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:29.357 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:29.357 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:29.358 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:29.359 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:29.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027630 con 0x7f63d8005e00 2021-03-09T10:16:29.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.346+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:29.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.346+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:16:29.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:16:29.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:29.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:29.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:30.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:30.363+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:30.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:30.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:30.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:30.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:30.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:30.513+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:31.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:31.512+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:31.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:31.512+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:32.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:32.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:32.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:32.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:33.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:33.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:33.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:33.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:34.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00196b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:34.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00196b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:34.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00196b0 secure :-1 s=READY pgs=14537 cs=0 l=1 rev1=1 rx=0x7f63e001d4e0 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:34.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00196b0 secure :-1 s=READY pgs=14537 cs=0 l=1 rev1=1 rx=0x7f63e001d4e0 tx=0x7f63f004de00).stop 2021-03-09T10:16:34.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:34.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:34.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00196b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:34.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00196b0 unknown :-1 s=CLOSED pgs=14537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:34.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:16:34.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:34.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:34.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:34.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:34.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:34.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:34.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:34.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8003400 2021-03-09T10:16:34.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:34.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc60 con 0x7f63d8009130 2021-03-09T10:16:34.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:34.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003840 2021-03-09T10:16:34.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:34.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:34.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.346+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:34.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.347+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:34.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.347+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:34.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.347+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:34.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.347+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:16:34.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.347+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:34.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.347+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:34.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.347+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:34.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.347+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:34.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.347+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:34.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:34.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:34.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:34.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:34.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:34.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:34.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:34.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:34.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:34.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:16:34.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:34.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:34.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00296a0 con 0x7f63d8003840 2021-03-09T10:16:34.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.348+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:34.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.349+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:34.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14555 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0023680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:34.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.349+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:16:34.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.349+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:34.361 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:34.351+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40252 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:34.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0023850 con 0x7f63d8003840 2021-03-09T10:16:34.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.352+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:34.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.352+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:34.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.352+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:34.363 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:34.363 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:34.363 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:34.364 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:34.364 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:34.364 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:34.364 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:34.364 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:34.365 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:34.365 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:34.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007be80 con 0x7f63d8003840 2021-03-09T10:16:34.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.355+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:34.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.355+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:16:34.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63d8003840 2021-03-09T10:16:34.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:34.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:34.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:35.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:35.364+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:35.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:35.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:35.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:35.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:36.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:36.513+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:36.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:36.513+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:37.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:37.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:37.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:37.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:38.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:38.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:38.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:38.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:39.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14555 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0023680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:39.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14555 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0023680).stop 2021-03-09T10:16:39.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:39.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:39.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:39.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=CLOSED pgs=14555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:39.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:16:39.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:39.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:39.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:39.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:39.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:39.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:39.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:39.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e5a0 con 0x7f63d8003400 2021-03-09T10:16:39.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:39.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021e20 con 0x7f63d8009130 2021-03-09T10:16:39.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:39.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:16:39.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:39.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:39.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:16:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:39.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:39.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:39.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:39.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:39.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:39.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:39.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:39.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:39.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:39.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:39.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:39.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00223f0 con 0x7f63d8005e00 2021-03-09T10:16:39.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:39.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:39.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:16:39.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:39.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.353+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:39.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.355+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:16:39.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.355+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14569 cs=0 l=1 rev1=1 rx=0x7f63f003e4e0 tx=0x7f63f001df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:39.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.355+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:39.367 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:39.357+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40290 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:39.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0039900 con 0x7f63d8005e00 2021-03-09T10:16:39.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.358+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:39.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.358+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:39.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.358+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:39.369 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:39.369 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:39.369 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:39.369 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:39.370 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:39.370 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:39.370 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:39.370 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:39.371 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:39.371 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:39.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.361+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63d8005e00 2021-03-09T10:16:39.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.361+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:39.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.361+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:16:39.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.363+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63d8005e00 2021-03-09T10:16:39.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:39.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:39.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:40.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:40.365+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:40.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:40.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:40.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:40.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:40.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:40.514+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:41.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:41.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:41.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:41.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:42.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:42.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:42.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:42.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:43.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:43.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:43.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:43.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:44.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:44.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:44.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14569 cs=0 l=1 rev1=1 rx=0x7f63f003e4e0 tx=0x7f63f001df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:44.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14569 cs=0 l=1 rev1=1 rx=0x7f63f003e4e0 tx=0x7f63f001df50).stop 2021-03-09T10:16:44.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:44.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:44.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:44.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:44.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:16:44.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:44.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:44.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:44.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:44.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:44.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:44.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:44.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb70 con 0x7f63d8003400 2021-03-09T10:16:44.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:44.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8009130 2021-03-09T10:16:44.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:44.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003840 2021-03-09T10:16:44.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:44.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:44.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.358+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:44.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.359+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:44.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.359+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:44.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.360+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:44.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.360+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:16:44.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.360+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:44.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.360+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:44.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.361+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:44.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.361+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:44.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.361+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:44.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:44.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:44.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:44.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:44.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:44.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:44.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:44.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:44.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:44.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00296a0 con 0x7f63d8003840 2021-03-09T10:16:44.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:44.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:44.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00223f0 con 0x7f63d8003840 2021-03-09T10:16:44.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:44.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.362+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:44.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.363+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14589 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:44.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.363+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:16:44.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.363+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:44.373 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:44.365+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40330 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:44.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8003840 2021-03-09T10:16:44.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.365+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:44.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.365+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:44.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.365+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:44.375 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:44.375 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:44.375 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:44.376 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:44.376 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:44.376 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:44.376 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:44.377 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:44.377 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:44.377 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:44.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.367+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8003840 2021-03-09T10:16:44.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.367+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:44.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.367+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:16:44.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f63d8003840 2021-03-09T10:16:44.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:44.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:44.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:45.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:45.366+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:45.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:45.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:45.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:45.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:46.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:46.514+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:46.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:46.514+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:47.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:47.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:47.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:47.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:48.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:48.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:48.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:48.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:49.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.366+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:16:49.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.366+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:49.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.366+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14589 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:49.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14589 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0000c30).stop 2021-03-09T10:16:49.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:49.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:49.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:49.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=CLOSED pgs=14589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:49.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:16:49.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:49.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:49.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:49.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:49.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:49.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:49.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:49.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8003400 2021-03-09T10:16:49.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:49.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00012d0 con 0x7f63d8009130 2021-03-09T10:16:49.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:49.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064f50 con 0x7f63d8005e00 2021-03-09T10:16:49.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:49.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:49.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:49.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:49.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.367+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:49.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.368+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:49.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.368+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:16:49.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.368+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:49.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.368+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:49.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.368+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:49.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.368+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:49.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.368+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:49.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.369+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:49.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:49.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:49.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:49.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:49.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:16:49.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:16:49.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:49.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:49.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00655e0 con 0x7f63d8005e00 2021-03-09T10:16:49.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:49.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:49.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00296a0 con 0x7f63d8005e00 2021-03-09T10:16:49.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:16:49.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:49.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14602 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0009600).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:49.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:16:49.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.370+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:49.382 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:49.371+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40368 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:49.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0023520 con 0x7f63d8005e00 2021-03-09T10:16:49.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.371+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:49.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.371+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:49.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.371+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:49.384 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:49.384 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:49.385 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:49.385 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:49.385 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:49.385 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:49.386 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:49.386 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:49.386 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:49.386 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:49.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:16:49.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.375+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:49.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.375+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:16:49.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63d8005e00 2021-03-09T10:16:49.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:49.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:49.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:50.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:50.367+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:50.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:50.515+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:50.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:50.515+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:50.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:50.515+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:51.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:51.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:51.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:51.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:52.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:52.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:52.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:52.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:53.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:53.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:53.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:53.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:54.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:54.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:54.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14602 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0009600).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:54.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14602 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0009600).stop 2021-03-09T10:16:54.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:54.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:54.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:54.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:54.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:16:54.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:54.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:54.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:54.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:54.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:54.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:54.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:54.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00252f0 con 0x7f63d8003400 2021-03-09T10:16:54.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:54.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:16:54.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:54.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:16:54.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:54.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:54.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:54.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:54.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.372+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:54.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.373+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:54.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.373+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:16:54.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.373+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:54.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.373+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:54.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.373+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:54.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.373+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:54.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.373+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:54.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:54.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:54.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:54.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:54.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:54.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:16:54.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:16:54.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:54.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:54.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e320 con 0x7f63d8003840 2021-03-09T10:16:54.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.374+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:54.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.375+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:54.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.375+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8003840 2021-03-09T10:16:54.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.375+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:16:54.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.375+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:54.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.375+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:16:54.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.375+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:54.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.375+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14613 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc058000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:54.387 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:54.377+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40408 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:54.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:16:54.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.378+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:54.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.378+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:54.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.378+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:54.389 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:54.389 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:54.389 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:54.389 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:54.390 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:54.390 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:54.390 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:54.390 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:54.390 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:54.391 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:54.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:16:54.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.380+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:54.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.380+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:16:54.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.381+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013d80 con 0x7f63d8003840 2021-03-09T10:16:54.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:54.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:54.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:55.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:55.368+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:16:55.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:55.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:55.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:55.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:56.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:56.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:56.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:56.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:57.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:57.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:57.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:57.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:58.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:58.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:58.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:58.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:59.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:16:59.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:16:59.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14613 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc058000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:16:59.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14613 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc058000).stop 2021-03-09T10:16:59.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:59.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:16:59.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:16:59.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=CLOSED pgs=14613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:16:59.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:16:59.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:59.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:16:59.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:59.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:16:59.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:16:59.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:16:59.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:59.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012930 con 0x7f63d8003400 2021-03-09T10:16:59.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:59.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8009130 2021-03-09T10:16:59.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:16:59.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:16:59.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:16:59.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:16:59.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:59.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:16:59.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.378+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:16:59.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.379+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:16:59.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.379+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:16:59.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.379+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:16:59.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.379+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:16:59.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.379+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:16:59.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.379+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:16:59.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.379+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:16:59.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.379+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:16:59.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:59.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:59.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:16:59.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:16:59.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:16:59.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:16:59.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:16:59.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:59.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:16:59.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:16:59.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:16:59.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e320 con 0x7f63d8005e00 2021-03-09T10:16:59.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:16:59.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:16:59.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:16:59.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14621 cs=0 l=1 rev1=1 rx=0x7f63fc051c40 tx=0x7f63fc06f290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:16:59.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.380+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:16:59.393 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:16:59.385+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40446 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:16:59.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc051cb0 con 0x7f63d8005e00 2021-03-09T10:16:59.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.386+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:16:59.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.386+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:16:59.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.386+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:16:59.395 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:16:59.395 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:16:59.395 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:59.396 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:16:59.396 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:16:59.396 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:16:59.396 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:16:59.396 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:16:59.397 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:16:59.397 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:16:59.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065d50 con 0x7f63d8005e00 2021-03-09T10:16:59.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.386+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:16:59.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.386+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:16:59.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63d8005e00 2021-03-09T10:16:59.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:16:59.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:16:59.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:00.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:00.369+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:00.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:00.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:00.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:00.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:00.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:00.516+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:01.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:01.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:01.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:01.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:02.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:02.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:02.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:02.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:03.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:03.516+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:03.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:03.516+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:04.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:04.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:04.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14621 cs=0 l=1 rev1=1 rx=0x7f63fc051c40 tx=0x7f63fc06f290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:04.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14621 cs=0 l=1 rev1=1 rx=0x7f63fc051c40 tx=0x7f63fc06f290).stop 2021-03-09T10:17:04.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:04.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:04.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:04.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:17:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:04.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:04.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:04.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:04.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:04.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d270 con 0x7f63d8003400 2021-03-09T10:17:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8009130 2021-03-09T10:17:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007be00 con 0x7f63d8003840 2021-03-09T10:17:04.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:04.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:04.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:04.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:04.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:17:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:04.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:04.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:04.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.386+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:04.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:04.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:04.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:04.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:04.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:04.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:17:04.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:17:04.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:04.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:04.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00296a0 con 0x7f63d8003840 2021-03-09T10:17:04.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:04.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:04.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00655e0 con 0x7f63d8003840 2021-03-09T10:17:04.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:17:04.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.387+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:04.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.388+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14634 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000dee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:04.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.388+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:17:04.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.388+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:04.400 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:04.392+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40486 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:04.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:17:04.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.392+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:04.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.392+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:04.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.392+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:04.402 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:04.402 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:04.402 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:04.403 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:04.403 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:04.403 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:04.403 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:04.403 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:04.404 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:04.404 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:04.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:17:04.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.392+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.392+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:17:04.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00603b0 con 0x7f63d8003840 2021-03-09T10:17:04.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:04.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:04.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:05.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:05.370+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:05.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:05.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:05.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:05.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:06.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:06.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:06.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:06.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:07.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:07.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:07.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:08.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:08.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:08.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:08.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:09.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:09.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14634 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000dee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:09.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14634 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f000dee0).stop 2021-03-09T10:17:09.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:09.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:09.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:09.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=CLOSED pgs=14634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:09.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:17:09.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:09.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:09.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:09.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:09.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:09.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:17:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00252f0 con 0x7f63d8009130 2021-03-09T10:17:09.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:17:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.392+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:09.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.393+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:09.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.393+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:09.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.393+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:09.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.393+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:17:09.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.393+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:09.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.393+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:09.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.393+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:09.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.393+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:09.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.393+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:09.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:09.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:09.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:09.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:17:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:17:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:09.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027c60 con 0x7f63d8005e00 2021-03-09T10:17:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:09.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00296a0 con 0x7f63d8005e00 2021-03-09T10:17:09.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:17:09.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:09.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14648 cs=0 l=1 rev1=1 rx=0x7f63f007bfe0 tx=0x7f63f007be90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:09.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:17:09.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.394+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:09.408 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:09.398+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40524 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:09.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007a460 con 0x7f63d8005e00 2021-03-09T10:17:09.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.399+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:09.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.399+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:09.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.399+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:09.410 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:09.410 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:09.410 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:09.410 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:09.411 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:09.411 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:09.411 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:09.411 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:09.412 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:09.412 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:09.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:17:09.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.403+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:09.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.403+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:17:09.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.404+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8005e00 2021-03-09T10:17:09.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:09.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:09.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:10.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:10.371+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:10.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:10.517+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:10.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:10.517+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:10.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:10.517+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:11.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:11.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:11.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:12.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:12.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:12.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:12.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:13.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:13.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:13.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:13.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:14.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:14.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:14.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14648 cs=0 l=1 rev1=1 rx=0x7f63f007bfe0 tx=0x7f63f007be90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:14.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14648 cs=0 l=1 rev1=1 rx=0x7f63f007bfe0 tx=0x7f63f007be90).stop 2021-03-09T10:17:14.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:14.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:14.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:14.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:14.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:17:14.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:14.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:14.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:14.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:14.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:14.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:14.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:14.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a930 con 0x7f63d8003400 2021-03-09T10:17:14.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:14.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a0d0 con 0x7f63d8009130 2021-03-09T10:17:14.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:14.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb30 con 0x7f63d8003840 2021-03-09T10:17:14.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:14.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:14.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:14.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:14.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.399+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:14.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:14.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:17:14.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:14.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:14.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:14.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:14.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:14.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:14.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:14.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:14.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:14.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:14.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:17:14.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:17:14.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:14.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:14.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00112d0 con 0x7f63d8003840 2021-03-09T10:17:14.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:14.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:14.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027c60 con 0x7f63d8003840 2021-03-09T10:17:14.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:17:14.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.400+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:14.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.401+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:17:14.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.401+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 secure :-1 s=READY pgs=14660 cs=0 l=1 rev1=1 rx=0x7f63f007cfe0 tx=0x7f63f00554a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:14.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.401+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:14.413 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:14.405+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40564 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:14.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0032570 con 0x7f63d8003840 2021-03-09T10:17:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.405+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.405+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.405+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:14.414 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:14.415 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:14.416 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:14.416 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:14.416 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:14.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:17:14.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.405+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:14.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.405+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:17:14.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.407+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:17:14.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:14.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:14.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:15.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:15.372+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:15.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:15.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:15.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:15.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:16.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:16.518+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:16.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:16.518+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:17.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:17.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:17.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:17.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:18.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:18.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:18.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:19.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:19.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:19.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 secure :-1 s=READY pgs=14660 cs=0 l=1 rev1=1 rx=0x7f63f007cfe0 tx=0x7f63f00554a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:19.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 secure :-1 s=READY pgs=14660 cs=0 l=1 rev1=1 rx=0x7f63f007cfe0 tx=0x7f63f00554a0).stop 2021-03-09T10:17:19.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:19.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:19.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:19.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 unknown :-1 s=CLOSED pgs=14660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:19.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:17:19.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:19.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:19.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:19.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:19.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:19.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:19.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:19.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:17:19.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:19.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8009130 2021-03-09T10:17:19.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:19.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bc50 con 0x7f63d8005e00 2021-03-09T10:17:19.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:19.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:19.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.405+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:19.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.406+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:19.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.406+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:19.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.406+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:19.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.406+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:17:19.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.406+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:19.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.406+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:19.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:19.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:19.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:19.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:19.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:19.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:19.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:19.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:19.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:17:19.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:17:19.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:19.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:19.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00223f0 con 0x7f63d8005e00 2021-03-09T10:17:19.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:19.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:19.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:17:19.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:17:19.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.407+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:19.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.408+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14679 cs=0 l=1 rev1=1 rx=0x7f63e0012cd0 tx=0x7f63fc071280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:19.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.408+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:17:19.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.408+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:19.420 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:19.411+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40602 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:19.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8005e00 2021-03-09T10:17:19.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.412+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:19.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.412+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:19.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.412+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:19.422 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:19.422 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:19.422 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:19.422 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:19.422 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:19.423 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:19.423 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:19.423 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:19.423 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:19.423 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:19.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:17:19.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.417+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:19.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.417+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:17:19.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065d50 con 0x7f63d8005e00 2021-03-09T10:17:19.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:19.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:19.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:20.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:20.373+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:20.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:20.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:20.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:20.519+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:21.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:21.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:21.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:21.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:22.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:22.519+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:22.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:22.519+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:23.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:23.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:23.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:23.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:24.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:24.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:24.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14679 cs=0 l=1 rev1=1 rx=0x7f63e0012cd0 tx=0x7f63fc071280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:24.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14679 cs=0 l=1 rev1=1 rx=0x7f63e0012cd0 tx=0x7f63fc071280).stop 2021-03-09T10:17:24.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:24.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:24.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:24.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:24.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:17:24.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:24.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:24.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:24.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:24.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:24.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:24.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:24.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:17:24.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:24.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8009130 2021-03-09T10:17:24.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:24.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:17:24.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:24.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:24.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:24.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:24.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:24.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:24.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:17:24.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:24.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.412+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:24.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.413+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:24.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.413+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:24.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.413+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:24.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:24.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:24.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:24.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:24.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:24.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:17:24.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:17:24.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:24.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:24.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00296a0 con 0x7f63d8003840 2021-03-09T10:17:24.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:24.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:24.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00223f0 con 0x7f63d8003840 2021-03-09T10:17:24.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:17:24.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.414+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:24.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.415+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 secure :-1 s=READY pgs=14697 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc0086a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:24.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.415+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:17:24.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.415+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:24.427 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:24.418+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40642 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:24.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f490 con 0x7f63d8003840 2021-03-09T10:17:24.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.419+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:24.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.419+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:24.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.419+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:24.428 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:24.428 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:24.429 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:24.429 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:24.429 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:24.429 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:24.430 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:24.430 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:24.430 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:24.430 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:24.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.420+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01f490 con 0x7f63d8003840 2021-03-09T10:17:24.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.420+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:24.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.420+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:17:24.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.421+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06fbf0 con 0x7f63d8003840 2021-03-09T10:17:24.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:24.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:24.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:25.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:25.374+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:25.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:25.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:25.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:25.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:26.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:26.520+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:26.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:26.520+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:27.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:27.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:27.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:27.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:28.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:28.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:28.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:28.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:29.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:29.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:29.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 secure :-1 s=READY pgs=14697 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc0086a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:29.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 secure :-1 s=READY pgs=14697 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc0086a0).stop 2021-03-09T10:17:29.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:29.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:29.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c790 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:29.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c790 unknown :-1 s=CLOSED pgs=14697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:29.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:17:29.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:29.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:29.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:29.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:29.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:29.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:29.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:29.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00119a0 con 0x7f63d8003400 2021-03-09T10:17:29.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:29.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00222b0 con 0x7f63d8009130 2021-03-09T10:17:29.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:29.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b360 con 0x7f63d8005e00 2021-03-09T10:17:29.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.420+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:29.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.420+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:29.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.420+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:29.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.420+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:29.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.420+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:29.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.420+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:29.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.420+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:17:29.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.421+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:29.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.421+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:29.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.421+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:29.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.421+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:29.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.421+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:29.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:29.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:29.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:29.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:29.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:29.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:17:29.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:17:29.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:29.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:29.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:17:29.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:29.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:29.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00112d0 con 0x7f63d8005e00 2021-03-09T10:17:29.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:17:29.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:29.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:17:29.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:29.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.422+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685c0 secure :-1 s=READY pgs=14703 cs=0 l=1 rev1=1 rx=0x7f63e007b390 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:29.434 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:29.425+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40680 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:29.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.427+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009550 con 0x7f63d8005e00 2021-03-09T10:17:29.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.427+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:29.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.427+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:29.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.427+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:29.436 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:29.436 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:29.436 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:29.436 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:29.436 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:29.437 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:29.437 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:29.437 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:29.437 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:29.438 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:29.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.429+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009550 con 0x7f63d8005e00 2021-03-09T10:17:29.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.429+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:29.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.429+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:17:29.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:17:29.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:29.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:29.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:30.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:30.375+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:30.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:30.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:30.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:30.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:30.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:30.521+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:31.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:31.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:31.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:31.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:32.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:32.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:32.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:32.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:33.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:33.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:33.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:33.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:34.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00685c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:34.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00685c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:34.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685c0 secure :-1 s=READY pgs=14703 cs=0 l=1 rev1=1 rx=0x7f63e007b390 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:34.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685c0 secure :-1 s=READY pgs=14703 cs=0 l=1 rev1=1 rx=0x7f63e007b390 tx=0x7f63f0000c30).stop 2021-03-09T10:17:34.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:34.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:34.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00685c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:34.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00685c0 unknown :-1 s=CLOSED pgs=14703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:17:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:34.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:34.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:17:34.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:34.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066280 con 0x7f63d8009130 2021-03-09T10:17:34.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:34.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:17:34.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:34.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:34.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:34.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:34.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:34.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:34.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:17:34.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:34.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.426+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:34.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:34.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:34.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:34.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:34.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:34.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:34.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:34.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:34.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:17:34.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:17:34.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:34.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:34.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bf90 con 0x7f63d8003840 2021-03-09T10:17:34.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:34.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:34.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00296a0 con 0x7f63d8003840 2021-03-09T10:17:34.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:17:34.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.427+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:34.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.428+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14714 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc075fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:34.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.428+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:17:34.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.428+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:34.441 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:34.433+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40718 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:34.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06dd90 con 0x7f63d8003840 2021-03-09T10:17:34.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.433+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:34.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.433+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:34.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.433+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:34.442 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:34.443 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:34.443 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:34.443 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:34.443 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:34.443 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:34.443 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:34.444 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:34.444 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:34.444 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:34.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013c60 con 0x7f63d8003840 2021-03-09T10:17:34.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.434+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:34.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.434+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:17:34.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.435+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012750 con 0x7f63d8003840 2021-03-09T10:17:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:34.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:35.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:35.376+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:35.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:35.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:35.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:35.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:36.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:36.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:36.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:36.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:37.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:37.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:37.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:37.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:38.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:38.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:38.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:38.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:39.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:39.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:39.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14714 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc075fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:39.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 secure :-1 s=READY pgs=14714 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc075fe0).stop 2021-03-09T10:17:39.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:39.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:39.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:39.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e240 unknown :-1 s=CLOSED pgs=14714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:39.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:17:39.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:39.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:39.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:39.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:39.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:39.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:39.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:39.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8003400 2021-03-09T10:17:39.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:39.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008f30 con 0x7f63d8009130 2021-03-09T10:17:39.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:39.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ec60 con 0x7f63d8005e00 2021-03-09T10:17:39.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:39.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:39.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:39.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:39.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.434+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:39.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.435+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:39.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.435+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:17:39.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.435+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:39.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.435+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:39.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.435+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:39.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.435+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:39.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.435+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:39.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.435+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:39.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:39.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:39.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:39.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:39.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:17:39.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:17:39.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:39.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:39.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00296a0 con 0x7f63d8005e00 2021-03-09T10:17:39.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:39.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:39.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:17:39.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:17:39.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:39.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.436+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14730 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f007bf70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:39.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.437+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:17:39.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.437+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:39.449 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:39.440+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40758 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:39.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:17:39.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.440+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:39.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.440+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:39.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.440+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:39.451 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:39.451 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:39.451 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:39.451 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:39.452 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:39.452 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:39.452 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:39.452 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:39.452 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:39.453 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:39.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:17:39.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.443+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:39.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.443+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:17:39.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.445+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8005e00 2021-03-09T10:17:39.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.521+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:39.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:39.521+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:40.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:40.377+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:40.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:40.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:40.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:40.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:40.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:40.522+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:41.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:41.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:41.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:41.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:42.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:42.522+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:42.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:42.522+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:43.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:43.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:43.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:43.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:44.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:44.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:44.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14730 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f007bf70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:44.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14730 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f007bf70).stop 2021-03-09T10:17:44.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:44.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:44.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:44.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:44.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:17:44.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:44.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:44.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:44.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:44.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:44.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:44.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:44.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb30 con 0x7f63d8003400 2021-03-09T10:17:44.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:44.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:17:44.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:44.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003840 2021-03-09T10:17:44.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:44.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:44.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:44.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:44.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.441+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:44.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.442+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:44.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.442+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:17:44.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.442+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:44.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.442+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:44.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.442+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:44.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.442+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:44.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.442+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:44.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:44.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:44.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:44.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:44.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:44.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:17:44.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:17:44.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:44.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:44.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b70 con 0x7f63d8003840 2021-03-09T10:17:44.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:44.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:44.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00296a0 con 0x7f63d8003840 2021-03-09T10:17:44.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:17:44.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.443+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:44.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.444+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e150 secure :-1 s=READY pgs=14743 cs=0 l=1 rev1=1 rx=0x7f63f000df20 tx=0x7f63f0038d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:44.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.444+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:17:44.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.444+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:44.456 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:44.447+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40798 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:44.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.447+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036380 con 0x7f63d8003840 2021-03-09T10:17:44.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.447+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:44.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.447+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:44.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.447+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:44.457 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:44.458 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:44.458 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:44.458 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:44.458 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:44.459 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:44.459 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:44.459 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:44.459 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:44.459 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:44.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00367b0 con 0x7f63d8003840 2021-03-09T10:17:44.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.448+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:44.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.448+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:17:44.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.449+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8003840 2021-03-09T10:17:44.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:44.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:44.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:45.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:45.378+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:45.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:45.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:45.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:45.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:46.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:46.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:46.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:46.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:47.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:47.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:47.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:47.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:48.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:48.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:48.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:48.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:49.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:49.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:49.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e150 secure :-1 s=READY pgs=14743 cs=0 l=1 rev1=1 rx=0x7f63f000df20 tx=0x7f63f0038d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:49.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e150 secure :-1 s=READY pgs=14743 cs=0 l=1 rev1=1 rx=0x7f63f000df20 tx=0x7f63f0038d60).stop 2021-03-09T10:17:49.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:49.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:49.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:49.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e150 unknown :-1 s=CLOSED pgs=14743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:49.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:17:49.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:49.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:49.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:49.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:49.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:49.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:49.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:49.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c730 con 0x7f63d8003400 2021-03-09T10:17:49.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:49.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:17:49.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:49.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a40 con 0x7f63d8005e00 2021-03-09T10:17:49.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:49.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:49.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:49.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:49.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:49.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:49.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:17:49.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:49.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.448+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:49.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.449+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:49.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.449+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:49.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.449+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:49.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:49.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:49.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:49.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:49.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:49.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:17:49.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:17:49.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:49.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:49.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001c530 con 0x7f63d8005e00 2021-03-09T10:17:49.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:49.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:49.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b70 con 0x7f63d8005e00 2021-03-09T10:17:49.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:17:49.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:49.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.450+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14763 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63f0038e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:49.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.451+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:17:49.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.451+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:49.462 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:49.454+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40836 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:49.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.454+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036380 con 0x7f63d8005e00 2021-03-09T10:17:49.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.454+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:49.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.454+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:49.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.454+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:49.464 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:49.464 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:49.464 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:49.464 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:49.464 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:49.465 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:49.465 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:49.465 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:49.465 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:49.465 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:49.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8005e00 2021-03-09T10:17:49.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.458+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:49.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.458+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:17:49.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8005e00 2021-03-09T10:17:49.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:49.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:49.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:50.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:50.379+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:50.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:50.523+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:50.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:50.523+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:50.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:50.523+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:51.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:51.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:51.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:51.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:52.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:52.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:52.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:52.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:53.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:53.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:53.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:53.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:54.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:17:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14763 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63f0038e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:54.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14763 cs=0 l=1 rev1=1 rx=0x7f63e0045750 tx=0x7f63f0038e80).stop 2021-03-09T10:17:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:54.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:17:54.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:54.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:54.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00014d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:54.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:54.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8003400 2021-03-09T10:17:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026f60 con 0x7f63d8009130 2021-03-09T10:17:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00134d0 con 0x7f63d8003840 2021-03-09T10:17:54.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:54.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:54.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00014d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:54.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:17:54.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:54.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:54.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:54.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:54.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.455+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:54.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:54.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:54.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:54.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:54.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:54.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:17:54.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:17:54.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:54.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:54.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e320 con 0x7f63d8003840 2021-03-09T10:17:54.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:54.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:54.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bf90 con 0x7f63d8003840 2021-03-09T10:17:54.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:17:54.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.456+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:54.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.457+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00014d0 secure :-1 s=READY pgs=14772 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc065240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:54.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.457+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:17:54.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.457+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:54.470 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:54.460+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40876 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:54.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc062270 con 0x7f63d8003840 2021-03-09T10:17:54.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.461+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:54.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.461+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:54.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.461+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:54.471 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:54.471 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:54.472 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:54.472 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:54.472 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:54.472 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:54.473 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:54.473 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:54.473 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:54.473 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:54.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.465+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:17:54.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.465+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:54.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.465+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:17:54.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc062270 con 0x7f63d8003840 2021-03-09T10:17:54.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:54.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:54.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:55.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:55.380+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:17:55.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:55.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:55.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:55.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:56.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:56.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:56.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:56.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:57.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:57.524+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:57.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:57.524+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:58.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:58.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:58.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:58.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:59.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:17:59.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00014d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:17:59.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00014d0 secure :-1 s=READY pgs=14772 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc065240).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:17:59.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00014d0 secure :-1 s=READY pgs=14772 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc065240).stop 2021-03-09T10:17:59.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:17:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00014d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:17:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00014d0 unknown :-1 s=CLOSED pgs=14772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:17:59.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:17:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:17:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:17:59.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:17:59.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:17:59.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:59.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:17:59.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:59.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:17:59.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:17:59.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:17:59.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:17:59.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:17:59.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:59.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:17:59.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.462+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:17:59.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.463+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:17:59.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.463+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:17:59.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.463+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:17:59.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.463+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:17:59.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.463+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:17:59.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.463+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:17:59.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.463+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:17:59.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:17:59.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:59.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:59.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:17:59.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:17:59.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:17:59.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:17:59.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:17:59.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:59.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:17:59.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:17:59.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:17:59.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e320 con 0x7f63d8005e00 2021-03-09T10:17:59.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:17:59.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.464+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:17:59.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.465+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:17:59.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.465+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:17:59.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.465+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002ed30 secure :-1 s=READY pgs=14787 cs=0 l=1 rev1=1 rx=0x7f63fc058360 tx=0x7f63fc0583d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:17:59.476 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:17:59.467+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40914 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:17:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.469+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012ca0 con 0x7f63d8005e00 2021-03-09T10:17:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.469+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:17:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.469+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:17:59.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.469+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:17:59.478 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:17:59.478 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:17:59.478 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:59.478 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:17:59.479 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:17:59.479 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:17:59.479 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:17:59.479 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:17:59.479 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:17:59.480 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:17:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.472+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:17:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.472+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:17:59.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.472+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:17:59.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012ca0 con 0x7f63d8005e00 2021-03-09T10:17:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:17:59.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:17:59.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:00.381+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:00.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:00.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:00.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:00.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:00.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:00.525+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:01.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:01.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:01.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:01.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:02.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:02.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:02.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:02.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:03.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:03.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:03.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:03.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:04.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e002ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:18:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e002ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002ed30 secure :-1 s=READY pgs=14787 cs=0 l=1 rev1=1 rx=0x7f63fc058360 tx=0x7f63fc0583d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:04.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002ed30 secure :-1 s=READY pgs=14787 cs=0 l=1 rev1=1 rx=0x7f63fc058360 tx=0x7f63fc0583d0).stop 2021-03-09T10:18:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e002ed30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:04.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e002ed30 unknown :-1 s=CLOSED pgs=14787 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:18:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:04.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:04.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:04.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:04.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:04.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:04.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ec60 con 0x7f63d8003400 2021-03-09T10:18:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019130 con 0x7f63d8009130 2021-03-09T10:18:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:04.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:18:04.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:04.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:04.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:04.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:04.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.468+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.469+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.469+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:18:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.469+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.469+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:04.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.469+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:04.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.469+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:04.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.469+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:04.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:04.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:18:04.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:18:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018b70 con 0x7f63d8003840 2021-03-09T10:18:04.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001c530 con 0x7f63d8003840 2021-03-09T10:18:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:18:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.470+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:04.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.471+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d630 secure :-1 s=READY pgs=14800 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f00887e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:04.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.472+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:18:04.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.472+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:04.483 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:04.474+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40954 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:04.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.475+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8003840 2021-03-09T10:18:04.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.475+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.475+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.475+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:04.485 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:04.486 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:04.487 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:04.487 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:04.487 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:04.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.479+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8003840 2021-03-09T10:18:04.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.479+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:04.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.479+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:18:04.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.479+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:18:04.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:04.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:04.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:05.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:05.382+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:05.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:05.525+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:05.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:05.525+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:06.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:06.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:06.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:07.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:07.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:07.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:07.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:08.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:08.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:08.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:08.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:09.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:18:09.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:09.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d630 secure :-1 s=READY pgs=14800 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f00887e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:09.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d630 secure :-1 s=READY pgs=14800 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f00887e0).stop 2021-03-09T10:18:09.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:09.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:09.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:09.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d630 unknown :-1 s=CLOSED pgs=14800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:09.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:18:09.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:09.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:09.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:09.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:09.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016330 con 0x7f63d8003400 2021-03-09T10:18:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bd20 con 0x7f63d8009130 2021-03-09T10:18:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00134d0 con 0x7f63d8005e00 2021-03-09T10:18:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:09.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:09.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.474+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:09.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:09.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.475+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:09.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.475+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:09.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.475+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:18:09.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.475+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:09.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.475+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:09.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:09.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:09.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:09.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:09.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:09.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:09.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:18:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:18:09.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:09.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:09.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T10:18:09.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:09.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:09.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018b70 con 0x7f63d8005e00 2021-03-09T10:18:09.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:18:09.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.476+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:09.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.477+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:18:09.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.477+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:09.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.477+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14815 cs=0 l=1 rev1=1 rx=0x7f63e003a830 tx=0x7f63f004c260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:09.489 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:09.480+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 40992 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:09.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.481+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:18:09.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.481+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:09.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.481+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:09.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.481+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:09.491 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:09.491 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:09.491 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:09.492 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:09.493 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:09.493 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:09.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.481+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:18:09.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.481+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:09.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.483+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029bd0 con 0x7f63d8005e00 2021-03-09T10:18:09.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.484+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:18:09.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:09.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:09.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:10.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:10.383+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:10.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:10.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:10.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:10.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:10.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:10.526+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:11.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:11.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:11.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:11.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:12.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:12.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:12.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:12.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:13.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:13.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:13.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:13.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:14.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:14.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:14.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14815 cs=0 l=1 rev1=1 rx=0x7f63e003a830 tx=0x7f63f004c260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:14.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14815 cs=0 l=1 rev1=1 rx=0x7f63e003a830 tx=0x7f63f004c260).stop 2021-03-09T10:18:14.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:14.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:14.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:14.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:14.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:18:14.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:14.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:14.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.480+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:14.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:14.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:14.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:14.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:14.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b6c0 con 0x7f63d8003400 2021-03-09T10:18:14.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:14.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:18:14.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:14.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d070 con 0x7f63d8003840 2021-03-09T10:18:14.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:14.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:14.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:14.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:14.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.481+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:14.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.482+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:14.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.482+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:18:14.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.482+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:14.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.482+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:14.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.482+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:14.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.482+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:14.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.482+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:14.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:14.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:14.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:14.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:14.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:14.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:18:14.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:18:14.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:14.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:14.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013ca0 con 0x7f63d8003840 2021-03-09T10:18:14.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:14.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:14.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:18:14.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:18:14.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:14.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:18:14.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 secure :-1 s=READY pgs=14833 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc075f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:14.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.483+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:14.496 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:14.486+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41031 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:14.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:18:14.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.487+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:14.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.487+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:14.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.487+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:14.497 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:14.498 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:14.498 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:14.498 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:14.498 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:14.499 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:14.499 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:14.499 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:14.499 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:14.499 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:14.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:18:14.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.491+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:14.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.491+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:18:14.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8003840 2021-03-09T10:18:14.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:14.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:14.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:15.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:15.384+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:15.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:15.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:15.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:15.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:16.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:16.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:16.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:16.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:17.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:17.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:17.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:17.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:18.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:18.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:18.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:18.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:19.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0025e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:19.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0025e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:19.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 secure :-1 s=READY pgs=14833 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc075f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:19.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 secure :-1 s=READY pgs=14833 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc075f70).stop 2021-03-09T10:18:19.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:19.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:19.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0025e10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:19.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0025e10 unknown :-1 s=CLOSED pgs=14833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:19.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:18:19.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:19.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:19.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:19.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:19.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:19.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:19.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:19.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002cd0 con 0x7f63d8003400 2021-03-09T10:18:19.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:19.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ec60 con 0x7f63d8009130 2021-03-09T10:18:19.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:19.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:18:19.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:19.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:19.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:19.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:19.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:19.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.488+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:19.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.488+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:18:19.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.488+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:19.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.488+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:19.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.488+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:19.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.488+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:19.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.488+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:19.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:19.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:19.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:19.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:19.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:19.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:18:19.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:18:19.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:19.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:19.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bf90 con 0x7f63d8005e00 2021-03-09T10:18:19.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:19.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:19.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013ca0 con 0x7f63d8005e00 2021-03-09T10:18:19.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:18:19.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.489+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:19.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.490+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14844 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc013f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:19.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.490+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:18:19.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.490+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:19.502 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:19.493+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41070 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:19.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T10:18:19.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.494+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:19.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.494+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:19.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.494+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:19.504 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:19.505 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:19.505 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:19.505 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:19.505 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:19.505 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:19.506 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:19.506 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:19.506 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:19.506 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:19.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8005e00 2021-03-09T10:18:19.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.494+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:19.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.494+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:18:19.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.496+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012ca0 con 0x7f63d8005e00 2021-03-09T10:18:19.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:19.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:19.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:20.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:20.385+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:20.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:20.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:20.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:20.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:20.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:20.526+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:21.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:21.526+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:21.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:21.526+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:22.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:22.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:22.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:22.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:23.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:23.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:23.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:23.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:24.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:24.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:24.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14844 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc013f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:24.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=14844 cs=0 l=1 rev1=1 rx=0x7f63fc00c010 tx=0x7f63fc013f00).stop 2021-03-09T10:18:24.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:24.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:24.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:24.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=14844 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:24.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:18:24.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:24.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:24.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:24.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:24.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:24.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:24.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:24.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:18:24.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:24.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:18:24.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:24.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a930 con 0x7f63d8003840 2021-03-09T10:18:24.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.492+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:24.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.493+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:24.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.493+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:24.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.493+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:24.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.493+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:24.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.494+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:24.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.494+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:18:24.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.494+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:24.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.494+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:24.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.494+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:24.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.494+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:24.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.494+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:24.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:24.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:24.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:24.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:24.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:24.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:18:24.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:18:24.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:24.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:24.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00296a0 con 0x7f63d8003840 2021-03-09T10:18:24.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:24.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:24.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8003840 2021-03-09T10:18:24.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:18:24.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:24.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:18:24.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:24.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.495+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14859 cs=0 l=1 rev1=1 rx=0x7f63e00327c0 tx=0x7f63f004c520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:24.507 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:24.498+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41108 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:24.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.501+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:18:24.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.501+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:24.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.501+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:24.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.501+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:24.509 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:24.509 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:24.509 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:24.509 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:24.509 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:24.510 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:24.510 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:24.510 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:24.510 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:24.511 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:24.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.501+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:18:24.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.501+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:24.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.501+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:18:24.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:18:24.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:24.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:24.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:25.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:25.385+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:25.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:25.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:25.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:25.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:26.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:26.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:26.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:26.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:27.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:27.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:27.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:27.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:28.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:28.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:28.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:28.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:29.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e0020e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:18:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e0020e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0020e50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4005d50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:29.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0020e50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4005d50).stop 2021-03-09T10:18:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e0020e50 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:18:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e0020e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0020e50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f6405551700 10 monclient: get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:18:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4050000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.348+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:18:29.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:29.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:29.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14859 cs=0 l=1 rev1=1 rx=0x7f63e00327c0 tx=0x7f63f004c520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:29.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14859 cs=0 l=1 rev1=1 rx=0x7f63e00327c0 tx=0x7f63f004c520).stop 2021-03-09T10:18:29.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:29.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:29.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:29.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=CLOSED pgs=14859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:29.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:18:29.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:29.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:29.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:29.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:29.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:29.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:29.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:29.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e4f0 con 0x7f63d8003400 2021-03-09T10:18:29.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:29.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:18:29.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:29.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d150 con 0x7f63e0018390 2021-03-09T10:18:29.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:29.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:29.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:29.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:29.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.499+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:29.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.500+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:29.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.500+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:18:29.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.500+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:29.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.500+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:29.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.500+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:29.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.500+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:29.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.500+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:29.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:29.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:29.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:29.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:29.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:29.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:18:29.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:18:29.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:29.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:29.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:18:29.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:29.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:29.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00296a0 con 0x7f63e0018390 2021-03-09T10:18:29.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:18:29.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.501+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:29.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.502+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e50 secure :-1 s=READY pgs=14875 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007f700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:29.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.502+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:18:29.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.502+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:29.517 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:29.505+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41158 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:29.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63e0018390 2021-03-09T10:18:29.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.506+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:29.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.506+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:29.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.506+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:29.519 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:29.519 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:29.519 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:29.520 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:29.520 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:29.520 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:29.520 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:29.521 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:29.521 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:29.521 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:29.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.508+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007f8c0 con 0x7f63e0018390 2021-03-09T10:18:29.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.508+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:29.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.508+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:18:29.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.509+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63e0018390 2021-03-09T10:18:29.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.527+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:29.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:29.527+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:30.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:30.386+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:30.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:30.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:30.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:30.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:30.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:30.528+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:31.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:31.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:31.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:31.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:32.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:32.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:32.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:32.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:33.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:33.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:33.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:33.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:34.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:34.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:34.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e50 secure :-1 s=READY pgs=14875 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007f700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:34.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e50 secure :-1 s=READY pgs=14875 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007f700).stop 2021-03-09T10:18:34.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:34.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:34.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:34.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020e50 unknown :-1 s=CLOSED pgs=14875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:34.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:18:34.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:34.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:34.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:34.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:34.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:34.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:34.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:34.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024ce0 con 0x7f63d8003400 2021-03-09T10:18:34.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:34.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007ab30 con 0x7f63d8009130 2021-03-09T10:18:34.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:34.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:18:34.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:34.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:34.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:34.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:34.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:34.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.507+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:34.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.507+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:18:34.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.507+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:34.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.507+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:34.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.507+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:34.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.507+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:34.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.507+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:34.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.507+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:34.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:34.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:34.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:34.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:34.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:18:34.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:18:34.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:34.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:34.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025c30 con 0x7f63d8003840 2021-03-09T10:18:34.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:34.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bf90 con 0x7f63d8003840 2021-03-09T10:18:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:18:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:34.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:18:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:34.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.508+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14888 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:34.523 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:34.511+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41196 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:34.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:18:34.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.511+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:34.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.511+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:34.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.511+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:34.525 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:34.525 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:34.525 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:34.526 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:34.526 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:34.526 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:34.526 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:34.527 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:34.527 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:34.527 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:34.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:18:34.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.514+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:34.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.514+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:18:34.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.515+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f570 con 0x7f63d8003840 2021-03-09T10:18:34.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:34.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:34.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:35.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:35.387+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:35.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:35.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:35.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:35.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:36.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:36.528+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:36.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:36.528+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:37.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:37.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:37.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:37.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:38.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:38.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:38.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:38.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:39.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:39.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:39.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14888 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:39.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14888 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc04a740).stop 2021-03-09T10:18:39.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:39.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:39.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:39.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=CLOSED pgs=14888 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:39.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:18:39.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:39.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:39.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:39.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d070 con 0x7f63d8003400 2021-03-09T10:18:39.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:39.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008f30 con 0x7f63d8009130 2021-03-09T10:18:39.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:39.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067f90 con 0x7f63e0018390 2021-03-09T10:18:39.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:39.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:39.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:39.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:39.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.512+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:39.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.513+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:39.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.513+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:18:39.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.513+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:39.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.513+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:39.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.513+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:39.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.513+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:39.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.513+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:39.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:39.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:39.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:39.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:39.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:39.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:18:39.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:18:39.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:39.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:39.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013ca0 con 0x7f63e0018390 2021-03-09T10:18:39.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:39.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:39.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025c30 con 0x7f63e0018390 2021-03-09T10:18:39.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:18:39.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:39.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020730 secure :-1 s=READY pgs=14900 cs=0 l=1 rev1=1 rx=0x7f63e0037ed0 tx=0x7f63fc06fcf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:39.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:18:39.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.514+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:39.527 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:39.519+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41236 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:39.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc008c90 con 0x7f63e0018390 2021-03-09T10:18:39.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.519+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:39.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.519+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:39.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.519+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:39.529 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:39.529 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:39.529 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:39.530 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:39.530 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:39.530 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:39.530 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:39.530 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:39.531 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:39.531 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:39.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc008c90 con 0x7f63e0018390 2021-03-09T10:18:39.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.519+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:39.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.520+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:18:39.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.521+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:18:39.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:39.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:39.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:40.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:40.388+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:40.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:40.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:40.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:40.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:40.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:40.529+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:41.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:41.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:41.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:41.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:42.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:42.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:42.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:42.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:43.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:43.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:43.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:43.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:44.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:44.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:44.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020730 secure :-1 s=READY pgs=14900 cs=0 l=1 rev1=1 rx=0x7f63e0037ed0 tx=0x7f63fc06fcf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:44.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020730 secure :-1 s=READY pgs=14900 cs=0 l=1 rev1=1 rx=0x7f63e0037ed0 tx=0x7f63fc06fcf0).stop 2021-03-09T10:18:44.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:44.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:44.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:44.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0020730 unknown :-1 s=CLOSED pgs=14900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:44.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:18:44.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:44.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:44.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:44.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:44.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:44.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:44.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:44.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b200 con 0x7f63d8003400 2021-03-09T10:18:44.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:44.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb30 con 0x7f63d8009130 2021-03-09T10:18:44.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:44.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:18:44.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:44.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:44.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:44.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:44.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.519+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:44.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.520+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:44.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.520+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:18:44.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.520+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:44.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.520+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:44.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:44.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:44.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:44.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:44.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:44.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:44.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:44.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:44.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:18:44.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:18:44.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:44.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:44.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d840 con 0x7f63d8003840 2021-03-09T10:18:44.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:44.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:44.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013ca0 con 0x7f63d8003840 2021-03-09T10:18:44.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:18:44.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.521+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:44.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.522+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14915 cs=0 l=1 rev1=1 rx=0x7f63fc02e070 tx=0x7f63fc058280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:44.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.522+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:18:44.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.522+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:44.534 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:44.525+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41274 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:44.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.526+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc008c90 con 0x7f63d8003840 2021-03-09T10:18:44.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.526+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:44.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.526+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:44.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.526+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:44.536 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:44.536 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:44.536 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:44.536 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:44.537 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:44.537 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:44.537 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:44.537 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:44.537 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:44.538 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:44.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:44.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:44.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.530+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc008c90 con 0x7f63d8003840 2021-03-09T10:18:44.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.530+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:44.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.530+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:18:44.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:44.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:18:45.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:45.389+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:45.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:45.529+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:45.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:45.529+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:46.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:46.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:46.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:46.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:47.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:47.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:47.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:47.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:48.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:48.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:48.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:48.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:49.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:18:49.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:49.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14915 cs=0 l=1 rev1=1 rx=0x7f63fc02e070 tx=0x7f63fc058280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:49.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14915 cs=0 l=1 rev1=1 rx=0x7f63fc02e070 tx=0x7f63fc058280).stop 2021-03-09T10:18:49.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=CLOSED pgs=14915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:18:49.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:49.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:49.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:49.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:49.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e005dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:49.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:49.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:49.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:18:49.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:49.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8009130 2021-03-09T10:18:49.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:49.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065870 con 0x7f63e0018390 2021-03-09T10:18:49.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:49.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:49.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.525+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:49.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.526+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:49.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.526+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:49.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.526+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e005dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:49.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.526+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:18:49.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.526+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:49.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.526+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:49.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.526+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:49.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.526+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:49.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.526+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:49.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:49.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:49.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:49.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:49.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:49.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:18:49.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:18:49.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:49.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:49.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00296a0 con 0x7f63e0018390 2021-03-09T10:18:49.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:49.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:49.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:18:49.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:18:49.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.527+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:49.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.528+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:18:49.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.528+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:49.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.528+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e005dcc0 secure :-1 s=READY pgs=14927 cs=0 l=1 rev1=1 rx=0x7f63f005f880 tx=0x7f63f0038f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:49.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:49.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:49.542 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:49.531+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41314 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:49.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.532+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63e0018390 2021-03-09T10:18:49.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.532+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:49.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.532+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:49.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.532+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:49.544 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:49.544 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:49.544 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:49.544 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:49.545 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:49.545 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:49.545 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:49.545 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:49.545 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:49.546 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:49.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.532+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T10:18:49.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.532+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:49.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.532+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:18:49.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:49.534+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00412d0 con 0x7f63e0018390 2021-03-09T10:18:50.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:50.390+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:50.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:50.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:50.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:50.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:50.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:50.530+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:51.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:51.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:51.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:51.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:52.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:52.530+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:52.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:52.530+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:53.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:53.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:53.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:53.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:54.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:54.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:54.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e005dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:54.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e005dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:54.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e005dcc0 secure :-1 s=READY pgs=14927 cs=0 l=1 rev1=1 rx=0x7f63f005f880 tx=0x7f63f0038f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:54.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e005dcc0 secure :-1 s=READY pgs=14927 cs=0 l=1 rev1=1 rx=0x7f63f005f880 tx=0x7f63f0038f60).stop 2021-03-09T10:18:54.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:54.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e005dcc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e005dcc0 unknown :-1 s=CLOSED pgs=14927 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:18:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:54.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:54.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:54.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:54.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:54.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:54.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:54.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:18:54.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:54.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:18:54.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:54.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e6f0 con 0x7f63d8003840 2021-03-09T10:18:54.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.531+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:54.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:54.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:54.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:54.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:54.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:54.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:18:54.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:54.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:54.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:54.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:54.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:54.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:54.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:54.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:54.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:54.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:54.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:18:54.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.532+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:18:54.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:54.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:54.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da60 con 0x7f63d8003840 2021-03-09T10:18:54.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:54.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:54.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00296a0 con 0x7f63d8003840 2021-03-09T10:18:54.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:18:54.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:54.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020ea0 secure :-1 s=READY pgs=14940 cs=0 l=1 rev1=1 rx=0x7f63f0038fd0 tx=0x7f63f00066f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:54.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:18:54.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.533+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:54.546 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:54.537+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41352 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:54.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:18:54.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.537+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:54.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.537+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:54.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.537+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:54.548 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:54.548 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:54.549 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:54.549 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:54.549 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:54.549 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:54.549 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:54.550 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:54.550 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:54.550 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:54.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:18:54.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.537+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:54.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.537+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:18:54.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:54.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63d8003840 2021-03-09T10:18:55.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:55.391+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:18:55.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:55.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:55.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:55.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:56.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:56.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:56.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:56.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:57.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:57.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:57.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:57.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:58.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:58.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:58.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:58.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:59.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:18:59.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:59.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0020ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:18:59.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0020ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:18:59.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020ea0 secure :-1 s=READY pgs=14940 cs=0 l=1 rev1=1 rx=0x7f63f0038fd0 tx=0x7f63f00066f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:18:59.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020ea0 secure :-1 s=READY pgs=14940 cs=0 l=1 rev1=1 rx=0x7f63f0038fd0 tx=0x7f63f00066f0).stop 2021-03-09T10:18:59.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:59.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:18:59.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0020ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:18:59.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020ea0 unknown :-1 s=CLOSED pgs=14940 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:18:59.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:18:59.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:59.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:18:59.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:59.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:18:59.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:18:59.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:18:59.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:59.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d6c0 con 0x7f63d8003400 2021-03-09T10:18:59.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:59.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d00 con 0x7f63d8009130 2021-03-09T10:18:59.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:18:59.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:18:59.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.537+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:18:59.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.538+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:18:59.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.538+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:59.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.538+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:18:59.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.538+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:18:59.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:18:59.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.539+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:18:59.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.539+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:18:59.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.539+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:18:59.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.539+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:18:59.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.539+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:18:59.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.539+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:18:59.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:18:59.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:59.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:59.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:18:59.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:18:59.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:18:59.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:18:59.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:18:59.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:59.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013ca0 con 0x7f63e0018390 2021-03-09T10:18:59.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:18:59.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:18:59.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T10:18:59.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:18:59.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.540+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:18:59.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.541+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065220 secure :-1 s=READY pgs=14957 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007be20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:18:59.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.541+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:18:59.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.541+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:18:59.554 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:18:59.543+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41392 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:18:59.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.544+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:18:59.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.544+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:18:59.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.544+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:18:59.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.544+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:18:59.556 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:18:59.556 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:18:59.556 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:59.556 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:18:59.556 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:18:59.557 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:18:59.557 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:18:59.557 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:18:59.557 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:18:59.558 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:18:59.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.544+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:18:59.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.544+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:18:59.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.544+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:18:59.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:18:59.545+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003e180 con 0x7f63e0018390 2021-03-09T10:19:00.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:00.392+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:00.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:00.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:00.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:00.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:00.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:00.531+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:01.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:01.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:01.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:02.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:02.531+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:02.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:02.531+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:03.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:03.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:03.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:03.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:04.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:04.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:04.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0065220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:04.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0065220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:04.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065220 secure :-1 s=READY pgs=14957 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007be20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:04.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065220 secure :-1 s=READY pgs=14957 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007be20).stop 2021-03-09T10:19:04.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:04.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:04.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0065220 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:04.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0065220 unknown :-1 s=CLOSED pgs=14957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:04.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:19:04.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:04.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.543+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:04.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:04.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:04.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:04.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:04.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:04.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a8f0 con 0x7f63d8003400 2021-03-09T10:19:04.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:04.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:19:04.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:04.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c120 con 0x7f63d8003840 2021-03-09T10:19:04.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:04.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:04.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:04.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:04.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:04.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:04.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:19:04.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:04.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.544+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:04.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:04.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:04.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:04.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:04.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:04.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:04.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:04.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:04.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:19:04.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:19:04.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:04.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:04.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bf90 con 0x7f63d8003840 2021-03-09T10:19:04.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:04.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:04.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013ca0 con 0x7f63d8003840 2021-03-09T10:19:04.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:19:04.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.545+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:04.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.547+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14968 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f007bc10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:04.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.547+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:19:04.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.547+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:04.572 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:04.549+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41430 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:04.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.550+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055e60 con 0x7f63d8003840 2021-03-09T10:19:04.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.550+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:04.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.550+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:04.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.550+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:04.575 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:04.575 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:04.576 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:04.577 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:04.577 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:04.577 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:04.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055e60 con 0x7f63d8003840 2021-03-09T10:19:04.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.551+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:04.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.551+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:19:04.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:04.552+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:19:05.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:05.393+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:05.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:05.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:05.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:05.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:06.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:06.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:06.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:06.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:07.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:07.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:07.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:07.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:08.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:08.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:08.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:08.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:09.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:09.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:09.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:09.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:09.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14968 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f007bc10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:09.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=14968 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f007bc10).stop 2021-03-09T10:19:09.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:09.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:09.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:09.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=CLOSED pgs=14968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:09.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:19:09.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0008650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:09.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:09.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:09.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:09.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:09.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:09.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:09.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d00 con 0x7f63d8003400 2021-03-09T10:19:09.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:09.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028720 con 0x7f63d8009130 2021-03-09T10:19:09.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d070 con 0x7f63e0018390 2021-03-09T10:19:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:09.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.550+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.551+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.551+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:19:09.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.551+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:09.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.551+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:09.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.551+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:09.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.551+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:09.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.551+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:09.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:09.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:09.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:09.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:09.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:09.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:19:09.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:19:09.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:09.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:09.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:19:09.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:09.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:09.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bf90 con 0x7f63e0018390 2021-03-09T10:19:09.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:19:09.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.552+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:09.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.553+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:19:09.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.553+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0008650 secure :-1 s=READY pgs=14984 cs=0 l=1 rev1=1 rx=0x7f63f0009a40 tx=0x7f63f003ec00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:09.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.553+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:09.565 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:09.556+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41468 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:09.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038dd0 con 0x7f63e0018390 2021-03-09T10:19:09.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.557+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.557+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.557+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:09.567 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:09.568 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:09.569 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:09.569 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:09.569 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:09.569 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:09.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.557+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0038dd0 con 0x7f63e0018390 2021-03-09T10:19:09.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.557+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:09.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.558+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:19:09.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:09.558+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007c350 con 0x7f63e0018390 2021-03-09T10:19:10.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:10.394+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:10.532+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:10.532+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:10.532+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:11.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:11.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:11.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:11.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:12.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:12.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:12.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:12.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:13.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:13.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:13.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:13.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:14.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:14.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:14.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:14.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:14.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0008650 secure :-1 s=READY pgs=14984 cs=0 l=1 rev1=1 rx=0x7f63f0009a40 tx=0x7f63f003ec00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:14.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0008650 secure :-1 s=READY pgs=14984 cs=0 l=1 rev1=1 rx=0x7f63f0009a40 tx=0x7f63f003ec00).stop 2021-03-09T10:19:14.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:14.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:14.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0008650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:14.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0008650 unknown :-1 s=CLOSED pgs=14984 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:14.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:19:14.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:14.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:14.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:14.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:14.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:14.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:14.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:14.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067f90 con 0x7f63d8003400 2021-03-09T10:19:14.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:14.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:19:14.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:14.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079890 con 0x7f63d8003840 2021-03-09T10:19:14.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.556+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:14.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.557+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:14.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.557+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:14.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.557+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:14.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.557+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:14.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.557+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:14.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.557+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:19:14.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.557+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:14.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.557+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:14.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:14.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:14.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:14.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:14.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:19:14.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:19:14.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:14.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:14.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024210 con 0x7f63d8003840 2021-03-09T10:19:14.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:19:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:19:14.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.558+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:14.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.559+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:19:14.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.559+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=14996 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0041e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:14.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.559+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:14.571 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:14.562+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41508 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:14.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:19:14.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.562+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.562+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.562+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:14.575 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:14.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.566+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:19:14.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.566+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:14.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.566+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:19:14.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:14.566+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:19:15.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:15.395+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:15.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:15.533+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:15.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:15.533+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:16.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:16.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:16.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:16.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:17.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:17.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:17.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:17.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:18.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:18.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:18.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:18.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:19.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:19.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:19.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:19.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=14996 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0041e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:19.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=14996 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0041e90).stop 2021-03-09T10:19:19.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:19.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:19.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:19.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=14996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:19.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:19:19.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:19.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:19.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:19.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:19.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:19.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:19.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:19.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:19:19.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:19.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d00 con 0x7f63d8009130 2021-03-09T10:19:19.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:19.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a10 con 0x7f63e0018390 2021-03-09T10:19:19.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:19.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024fd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:19.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.562+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:19.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.563+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:19.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.563+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:19.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.563+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:19.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:19:19.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:19.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:19.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:19.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:19.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:19.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:19.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:19.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:19.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:19.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:19.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:19:19.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:19:19.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:19.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:19.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00296a0 con 0x7f63e0018390 2021-03-09T10:19:19.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:19.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:19.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.564+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024210 con 0x7f63e0018390 2021-03-09T10:19:19.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.565+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:19:19.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.565+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:19.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.565+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15011 cs=0 l=1 rev1=1 rx=0x7f63f002f7a0 tx=0x7f63f007f5a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:19.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.565+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:19:19.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.565+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:19.577 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:19.569+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41546 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:19.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:19:19.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.573+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:19.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.573+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:19.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.573+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:19.579 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:19.579 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:19.579 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:19.580 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:19.580 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:19.580 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:19.580 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:19.581 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:19.581 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:19.581 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:19.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:19:19.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.573+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:19.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.573+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:19:19.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:19.574+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:19:20.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:20.396+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:20.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:20.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:20.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:20.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:20.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:20.534+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:21.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:21.534+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:21.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:21.534+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:22.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:22.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:22.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:22.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:23.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:23.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:23.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:23.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:24.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:24.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:24.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15011 cs=0 l=1 rev1=1 rx=0x7f63f002f7a0 tx=0x7f63f007f5a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:24.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15011 cs=0 l=1 rev1=1 rx=0x7f63f002f7a0 tx=0x7f63f007f5a0).stop 2021-03-09T10:19:24.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:24.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:24.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:24.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:24.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:19:24.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:24.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:24.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:24.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:24.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067f90 con 0x7f63d8003400 2021-03-09T10:19:24.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:24.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:19:24.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:24.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d00 con 0x7f63d8003840 2021-03-09T10:19:24.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.569+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:24.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.570+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:24.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.570+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:24.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.570+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:24.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.570+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:24.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.570+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:24.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.570+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:19:24.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.570+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:24.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.570+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:24.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:24.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:24.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:24.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:24.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:24.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:24.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:24.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:24.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:19:24.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:19:24.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:24.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:24.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bf90 con 0x7f63d8003840 2021-03-09T10:19:24.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:24.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:24.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00296a0 con 0x7f63d8003840 2021-03-09T10:19:24.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:19:24.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.571+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:24.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.572+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15023 cs=0 l=1 rev1=1 rx=0x7f63e0021880 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:24.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.572+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:19:24.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.572+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:24.585 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:24.575+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41586 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:24.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:19:24.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.576+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:24.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.576+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:24.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.576+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:24.587 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:24.587 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:24.587 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:24.587 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:24.587 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:24.588 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:24.588 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:24.588 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:24.588 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:24.588 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:24.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.577+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:19:24.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.577+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:24.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.577+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:19:24.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:24.578+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042320 con 0x7f63d8003840 2021-03-09T10:19:25.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:25.397+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:25.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:25.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:25.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:25.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:26.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:26.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:26.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:26.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:27.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:27.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:27.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:27.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:28.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:28.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:28.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:28.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:29.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:29.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:29.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:29.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:29.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15023 cs=0 l=1 rev1=1 rx=0x7f63e0021880 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:29.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15023 cs=0 l=1 rev1=1 rx=0x7f63e0021880 tx=0x7f63f004d180).stop 2021-03-09T10:19:29.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:29.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:29.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:29.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=15023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:29.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:19:29.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:29.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:29.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:29.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:29.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:29.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:29.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:29.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006df10 con 0x7f63d8003400 2021-03-09T10:19:29.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:29.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028720 con 0x7f63d8009130 2021-03-09T10:19:29.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:29.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ba80 con 0x7f63e0018390 2021-03-09T10:19:29.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:29.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:29.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:29.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:29.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.576+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:29.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:29.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:19:29.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:29.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:29.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:29.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:29.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:29.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:29.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:19:29.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:19:29.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:29.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:29.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d620 con 0x7f63e0018390 2021-03-09T10:19:29.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:29.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bf90 con 0x7f63e0018390 2021-03-09T10:19:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:19:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.577+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.578+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15036 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f004c2f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.578+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:19:29.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.578+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:29.592 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:29.581+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41626 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:29.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:19:29.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.582+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:29.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.582+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:29.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.582+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:29.594 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:29.594 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:29.594 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:29.594 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:29.595 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:29.595 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:29.595 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:29.595 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:29.595 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:29.596 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:29.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.586+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:19:29.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.586+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:29.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.587+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:19:29.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:29.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042db0 con 0x7f63e0018390 2021-03-09T10:19:30.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:30.398+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:30.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:30.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:30.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:30.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:30.536+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:31.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:31.535+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:31.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:31.535+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:32.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:32.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:32.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:32.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:33.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:33.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:33.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:33.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:34.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:34.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:34.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:34.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:34.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15036 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f004c2f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:34.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15036 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f004c2f0).stop 2021-03-09T10:19:34.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:19:34.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:34.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:34.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:34.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:34.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:34.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:34.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:34.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e640 con 0x7f63d8003400 2021-03-09T10:19:34.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:34.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:19:34.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:34.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015d00 con 0x7f63d8003840 2021-03-09T10:19:34.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:34.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:34.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:34.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:34.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.582+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:34.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.584+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:34.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.584+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:19:34.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.584+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:34.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.584+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:34.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.584+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:34.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.584+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:34.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.584+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:34.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:34.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:34.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:34.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:34.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:34.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:19:34.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:19:34.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:34.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:34.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025bc0 con 0x7f63d8003840 2021-03-09T10:19:34.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:34.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:34.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d840 con 0x7f63d8003840 2021-03-09T10:19:34.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:19:34.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.585+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:34.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.586+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15055 cs=0 l=1 rev1=1 rx=0x7f63e0014730 tx=0x7f63fc075fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:34.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.586+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:19:34.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.586+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:34.961 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:34.588+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41664 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:34.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065c10 con 0x7f63d8003840 2021-03-09T10:19:34.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.588+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:34.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.588+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:34.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.588+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:34.962 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:34.963 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:34.963 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:34.963 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:34.963 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:34.964 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:34.964 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:34.964 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:34.964 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:34.964 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:34.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065c10 con 0x7f63d8003840 2021-03-09T10:19:34.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.592+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:34.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:19:34.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:34.592+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:19:35.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:35.399+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:35.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:35.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:35.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:35.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:36.536+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:36.536+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:37.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:37.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:37.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:37.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:38.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:38.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:38.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:38.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:39.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:39.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:39.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:39.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:39.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15055 cs=0 l=1 rev1=1 rx=0x7f63e0014730 tx=0x7f63fc075fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:39.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15055 cs=0 l=1 rev1=1 rx=0x7f63e0014730 tx=0x7f63fc075fe0).stop 2021-03-09T10:19:39.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:39.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:39.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:39.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=15055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:39.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:19:39.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:39.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:39.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:39.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:39.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:39.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:39.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:39.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb30 con 0x7f63d8003400 2021-03-09T10:19:39.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:39.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:19:39.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:39.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023cb0 con 0x7f63e0018390 2021-03-09T10:19:39.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:39.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:39.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.588+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:39.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.589+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:39.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.589+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:39.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.589+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:39.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.589+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:19:39.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.589+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:39.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.589+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:39.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.590+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:39.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.590+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:39.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.590+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:39.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.590+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:39.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:39.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:39.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:39.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:39.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:19:39.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:19:39.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:39.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:39.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T10:19:39.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:39.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:39.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d620 con 0x7f63e0018390 2021-03-09T10:19:39.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:19:39.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.591+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:39.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.592+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:19:39.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.592+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:39.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.592+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15069 cs=0 l=1 rev1=1 rx=0x7f63f004d010 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:39.606 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:39.594+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41704 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:39.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.595+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T10:19:39.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.595+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:39.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.595+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:39.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.595+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:39.607 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:39.608 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:39.608 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:39.608 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:39.608 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:39.608 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:39.609 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:39.609 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:39.609 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:39.609 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:39.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.600+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T10:19:39.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.600+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:39.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.600+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ad80 con 0x7f63e0018390 2021-03-09T10:19:39.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:39.600+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:19:40.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:40.400+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:40.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:40.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:40.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:40.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:40.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:40.537+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:41.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:41.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:41.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:41.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:42.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:42.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:42.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:42.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:43.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:43.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:43.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:43.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:44.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:44.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:44.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:44.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:44.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15069 cs=0 l=1 rev1=1 rx=0x7f63f004d010 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:44.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15069 cs=0 l=1 rev1=1 rx=0x7f63f004d010 tx=0x7f63f004de00).stop 2021-03-09T10:19:44.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:44.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:44.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:19:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:44.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017b00 con 0x7f63d8003400 2021-03-09T10:19:44.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:44.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:19:44.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:44.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ea80 con 0x7f63d8003840 2021-03-09T10:19:44.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:44.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:44.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:44.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:44.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:19:44.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.595+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:44.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:44.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:19:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:19:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:19:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:44.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.596+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025bc0 con 0x7f63d8003840 2021-03-09T10:19:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.597+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:19:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.597+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.597+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c7d0 secure :-1 s=READY pgs=15076 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc066370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:44.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.597+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:19:44.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.597+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:44.611 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:44.601+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41742 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.602+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065e40 con 0x7f63d8003840 2021-03-09T10:19:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.602+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.602+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:44.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.602+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:44.613 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:44.614 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:44.615 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.606+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065e40 con 0x7f63d8003840 2021-03-09T10:19:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.606+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.606+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:19:44.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:44.606+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:19:45.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:45.401+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:45.537+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:45.537+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:46.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:46.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:47.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:47.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:47.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:47.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:48.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:48.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:48.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:48.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:49.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:49.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:49.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:19:49.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:49.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c7d0 secure :-1 s=READY pgs=15076 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc066370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:49.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c7d0 secure :-1 s=READY pgs=15076 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc066370).stop 2021-03-09T10:19:49.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:49.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:49.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:49.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c7d0 unknown :-1 s=CLOSED pgs=15076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:49.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:19:49.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:49.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:49.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:49.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:49.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:49.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:49.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:49.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:19:49.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:49.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001910 con 0x7f63d8009130 2021-03-09T10:19:49.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:49.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:19:49.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:49.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:49.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:49.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.601+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:49.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.602+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:49.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.602+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:49.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.602+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:19:49.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.602+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:49.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.602+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:49.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.603+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:49.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.603+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.603+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:49.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:49.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:49.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:49.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:19:49.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:19:49.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:49.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:49.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bf90 con 0x7f63e0018390 2021-03-09T10:19:49.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:49.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:49.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T10:19:49.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:19:49.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:49.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.604+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15093 cs=0 l=1 rev1=1 rx=0x7f63e005e340 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:49.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.605+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:19:49.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.605+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:49.617 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:49.607+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41782 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:49.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.607+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63e0018390 2021-03-09T10:19:49.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.607+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:49.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.607+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:49.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.607+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:49.619 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:49.619 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:49.619 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:49.620 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:49.620 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:49.620 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:49.620 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:49.620 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:49.621 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:49.621 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:49.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63e0018390 2021-03-09T10:19:49.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.610+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:49.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.610+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:19:49.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:49.612+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042320 con 0x7f63e0018390 2021-03-09T10:19:50.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:50.402+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:50.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:50.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:50.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:50.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:50.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:50.538+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:51.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:51.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:51.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:51.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:52.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:52.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:52.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:52.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:53.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:53.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:53.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:53.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:54.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:54.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:54.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:19:54.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:54.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15093 cs=0 l=1 rev1=1 rx=0x7f63e005e340 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:54.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15093 cs=0 l=1 rev1=1 rx=0x7f63e005e340 tx=0x7f63f004db20).stop 2021-03-09T10:19:54.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:54.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:54.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:54.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:54.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:19:54.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:54.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:54.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:54.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:54.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:54.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:54.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:54.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a40 con 0x7f63d8003400 2021-03-09T10:19:54.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:54.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ba80 con 0x7f63d8009130 2021-03-09T10:19:54.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:54.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8003840 2021-03-09T10:19:54.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:54.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:54.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:54.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:54.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:54.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:54.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:19:54.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:54.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.608+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:54.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:54.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:54.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:54.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:54.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:54.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:54.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024f30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:54.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:54.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:19:54.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:19:54.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:54.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:54.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c600 con 0x7f63d8003840 2021-03-09T10:19:54.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:54.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:54.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.610+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bf90 con 0x7f63d8003840 2021-03-09T10:19:54.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.611+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:19:54.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.611+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:54.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.611+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 secure :-1 s=READY pgs=15115 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007f6b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:54.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.611+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:19:54.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.611+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:54.625 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:54.614+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41820 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:54.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.614+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0057910 con 0x7f63d8003840 2021-03-09T10:19:54.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.614+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:54.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.614+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:54.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.614+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:54.627 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:54.627 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:54.627 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:54.628 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:54.628 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:54.629 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:54.629 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:54.630 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:54.630 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:54.631 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:54.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0032360 con 0x7f63d8003840 2021-03-09T10:19:54.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.618+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:54.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.618+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:19:54.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:54.620+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0057700 con 0x7f63d8003840 2021-03-09T10:19:55.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:55.403+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:19:55.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:55.538+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:55.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:55.538+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:56.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:56.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:56.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:56.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:57.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:57.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:57.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:57.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:58.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:58.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:58.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:58.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:59.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:19:59.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:59.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:19:59.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:19:59.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 secure :-1 s=READY pgs=15115 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007f6b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:19:59.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 secure :-1 s=READY pgs=15115 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007f6b0).stop 2021-03-09T10:19:59.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:59.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:19:59.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002f410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:19:59.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 unknown :-1 s=CLOSED pgs=15115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:19:59.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:19:59.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:59.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:19:59.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:59.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:19:59.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:19:59.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:19:59.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:59.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a190 con 0x7f63d8003400 2021-03-09T10:19:59.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:59.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:19:59.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:19:59.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00158b0 con 0x7f63e0018390 2021-03-09T10:19:59.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:19:59.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:19:59.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:59.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:19:59.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.615+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:19:59.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.616+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:19:59.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.616+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:19:59.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.616+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:19:59.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.616+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:19:59.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:19:59.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:19:59.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:19:59.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:19:59.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:19:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:19:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:19:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:19:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:19:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e480 con 0x7f63e0018390 2021-03-09T10:19:59.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:19:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:19:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:19:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:19:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.617+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:19:59.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.618+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:19:59.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.618+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:19:59.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.618+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15128 cs=0 l=1 rev1=1 rx=0x7f63e0030610 tx=0x7f63fc03fea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:19:59.629 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:19:59.620+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41860 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:19:59.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.621+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:19:59.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.621+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:19:59.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.621+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:19:59.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.621+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:19:59.631 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:19:59.631 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:19:59.631 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:59.631 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:19:59.632 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:19:59.632 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:19:59.632 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:19:59.632 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:19:59.633 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:19:59.633 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:19:59.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.626+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63e0018390 2021-03-09T10:19:59.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.626+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:19:59.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.626+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:19:59.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:19:59.626+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:20:00.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:00.404+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:00.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:00.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:00.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:00.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:00.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:00.540+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:01.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:01.539+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:01.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:01.539+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:02.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:02.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:02.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:02.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:03.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:03.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:03.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:04.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:04.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:04.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:20:04.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:04.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15128 cs=0 l=1 rev1=1 rx=0x7f63e0030610 tx=0x7f63fc03fea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:04.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15128 cs=0 l=1 rev1=1 rx=0x7f63e0030610 tx=0x7f63fc03fea0).stop 2021-03-09T10:20:04.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:20:04.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:04.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:04.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:04.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:04.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:04.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:04.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:04.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.621+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T10:20:04.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:04.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b6a0 con 0x7f63d8009130 2021-03-09T10:20:04.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:04.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:20:04.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:04.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024ef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:04.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:04.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:04.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:04.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:04.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:20:04.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:04.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:04.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:04.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:04.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.622+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:04.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:04.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:04.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:04.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024ef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:04.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:04.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:20:04.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:20:04.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:04.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:04.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:20:04.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:04.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:04.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e480 con 0x7f63d8003840 2021-03-09T10:20:04.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:20:04.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:04.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:20:04.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:04.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.623+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000af40 secure :-1 s=READY pgs=15137 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc023cc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:04.639 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:04.627+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41898 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:04.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013eb0 con 0x7f63d8003840 2021-03-09T10:20:04.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.628+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:04.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.628+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:04.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.628+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:04.640 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:04.640 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:04.641 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:04.641 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:04.641 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:04.641 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:04.642 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:04.642 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:04.642 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:04.642 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:04.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.630+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013eb0 con 0x7f63d8003840 2021-03-09T10:20:04.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.630+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:04.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.630+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:20:04.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:04.632+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63d8003840 2021-03-09T10:20:05.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:05.405+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:05.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:05.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:05.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:05.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:06.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:06.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:06.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:06.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:07.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:07.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:07.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:07.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:08.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:08.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:08.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:08.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:09.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:09.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:09.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:09.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:09.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000af40 secure :-1 s=READY pgs=15137 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc023cc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:09.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000af40 secure :-1 s=READY pgs=15137 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc023cc0).stop 2021-03-09T10:20:09.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:09.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:09.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:09.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000af40 unknown :-1 s=CLOSED pgs=15137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:09.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:20:09.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:09.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:09.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:09.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:09.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:09.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:09.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c310 con 0x7f63d8003400 2021-03-09T10:20:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b6a0 con 0x7f63d8009130 2021-03-09T10:20:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a10 con 0x7f63e0018390 2021-03-09T10:20:09.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:09.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:09.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:09.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:09.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:09.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.628+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:09.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.629+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:20:09.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.629+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:09.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.629+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:09.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:09.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:09.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:09.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:09.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:09.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:09.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:09.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:09.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:20:09.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:20:09.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:09.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:09.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027c30 con 0x7f63e0018390 2021-03-09T10:20:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c600 con 0x7f63e0018390 2021-03-09T10:20:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:20:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.630+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:09.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.631+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15156 cs=0 l=1 rev1=1 rx=0x7f63e0015cb0 tx=0x7f63f00272c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:09.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.631+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:20:09.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.631+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:09.644 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:09.634+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41938 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:09.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:20:09.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.635+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:09.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.635+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:09.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.635+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:09.646 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:09.647 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:09.647 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:09.647 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:09.647 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:09.648 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:09.648 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:09.648 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:09.648 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:09.648 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:09.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.636+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:20:09.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.636+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:09.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.636+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:20:09.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:09.637+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0039de0 con 0x7f63e0018390 2021-03-09T10:20:10.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:10.406+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:10.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:10.540+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:10.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:10.540+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:10.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:10.540+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:11.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:11.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:11.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:11.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:12.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:12.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:12.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:12.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:13.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:13.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:13.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:13.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:14.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:14.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:14.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:14.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:14.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15156 cs=0 l=1 rev1=1 rx=0x7f63e0015cb0 tx=0x7f63f00272c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:14.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15156 cs=0 l=1 rev1=1 rx=0x7f63e0015cb0 tx=0x7f63f00272c0).stop 2021-03-09T10:20:14.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:14.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:14.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:14.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:14.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:20:14.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:14.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:14.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:14.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:14.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:14.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:14.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:14.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8003400 2021-03-09T10:20:14.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:14.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c4c0 con 0x7f63d8009130 2021-03-09T10:20:14.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:14.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021260 con 0x7f63d8003840 2021-03-09T10:20:14.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:14.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:14.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.635+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:14.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.636+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:14.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.636+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:20:14.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.636+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:14.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.636+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:14.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.636+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:14.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.636+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:14.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:14.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:14.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:14.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:14.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:14.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:14.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:14.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:14.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:20:14.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:20:14.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:14.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:14.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0009700 con 0x7f63d8003840 2021-03-09T10:20:14.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:14.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:14.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027c30 con 0x7f63d8003840 2021-03-09T10:20:14.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.638+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:20:14.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.638+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:14.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.638+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2021-03-09T10:20:14.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.638+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021e00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f007bfa0).write client ident write failed r=-32 ((32) Broken pipe) 2021-03-09T10:20:14.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.638+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:14.650 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:14.641+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 41976 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:14.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:14.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6405d52700 10 monclient: get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:20:14.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6405d52700 0 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-03-09T10:20:14.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:14.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:14.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:14.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:14.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:14.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:20:14.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:14.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:14.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:14.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:14.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:14.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:14.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:14.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:20:14.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:14.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:20:14.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:14.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012a70 con 0x7f63e0018390 2021-03-09T10:20:14.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:14.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:14.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:14.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:14.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:20:14.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:14.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:14.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:14.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:14.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:14.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:14.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:14.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:14.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:14.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:14.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:14.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:14.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:20:14.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:20:14.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:14.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:14.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013210 con 0x7f63e0018390 2021-03-09T10:20:14.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:14.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:14.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:20:14.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:20:14.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:14.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15173 cs=0 l=1 rev1=1 rx=0x7f63e0043440 tx=0x7f63fc023be0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:14.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:20:14.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:14.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc07af10 con 0x7f63e0018390 2021-03-09T10:20:14.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:14.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:14.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:14.856 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:14.856 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:14.857 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:14.857 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:14.857 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:14.857 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:14.857 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:14.858 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:14.858 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:14.858 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:14.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013eb0 con 0x7f63e0018390 2021-03-09T10:20:14.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:14.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.839+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:20:14.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:14.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03cd90 con 0x7f63e0018390 2021-03-09T10:20:15.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:15.407+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:15.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:15.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:15.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:15.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:16.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:16.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:16.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:16.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:17.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:17.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:17.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:17.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:18.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:18.541+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:18.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:18.541+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:19.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:19.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:19.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:19.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:19.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15173 cs=0 l=1 rev1=1 rx=0x7f63e0043440 tx=0x7f63fc023be0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:19.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15173 cs=0 l=1 rev1=1 rx=0x7f63e0043440 tx=0x7f63fc023be0).stop 2021-03-09T10:20:19.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:19.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:19.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:19.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:19.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:20:19.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:19.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:19.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:19.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:19.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:19.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:19.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:19.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021260 con 0x7f63d8003400 2021-03-09T10:20:19.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:19.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c4c0 con 0x7f63d8009130 2021-03-09T10:20:19.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:19.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:20:19.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:19.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:19.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:19.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:19.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:19.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:19.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:20:19.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:19.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.642+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:19.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:19.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:19.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:19.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:19.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:19.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:19.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:19.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:19.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:20:19.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:20:19.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:19.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:19.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:20:19.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:19.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:19.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0009700 con 0x7f63d8003840 2021-03-09T10:20:19.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:20:19.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.643+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:19.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068190 secure :-1 s=READY pgs=15175 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007b4c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:19.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.644+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:20:19.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.644+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:19.658 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:19.647+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42016 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:19.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:20:19.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.648+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:19.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.648+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:19.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.648+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:19.659 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:19.660 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:19.660 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:19.660 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:19.660 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:19.661 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:19.661 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:19.661 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:19.661 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:19.661 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:19.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8003840 2021-03-09T10:20:19.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.648+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:19.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.649+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:20:19.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:19.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:20:20.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:20.408+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:20.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:20.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:20.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:20.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:20.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:20.542+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:21.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:21.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:21.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:21.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:22.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:22.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:22.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:22.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:23.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:23.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:23.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:23.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:24.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:24.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:24.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:24.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:24.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068190 secure :-1 s=READY pgs=15175 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007b4c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:24.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068190 secure :-1 s=READY pgs=15175 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007b4c0).stop 2021-03-09T10:20:24.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:24.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:24.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068190 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:24.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068190 unknown :-1 s=CLOSED pgs=15175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:24.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:20:24.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:24.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:24.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:24.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:24.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:24.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:24.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:24.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8003400 2021-03-09T10:20:24.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:24.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb30 con 0x7f63d8009130 2021-03-09T10:20:24.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:24.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d5c0 con 0x7f63e0018390 2021-03-09T10:20:24.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:24.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:24.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:24.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:24.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:24.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:24.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:20:24.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:24.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.648+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:24.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.649+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:24.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.649+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:24.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.649+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:24.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:24.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:24.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:24.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:24.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:24.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:20:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:20:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e480 con 0x7f63e0018390 2021-03-09T10:20:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:20:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:20:24.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15191 cs=0 l=1 rev1=1 rx=0x7f63f0027290 tx=0x7f63f007bfd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:20:24.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.650+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:24.662 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:24.653+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42054 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:24.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.655+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:20:24.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.655+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:24.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.655+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:24.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.655+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:24.664 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:24.664 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:24.665 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:24.666 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:24.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:20:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.658+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.658+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:20:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:24.659+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:20:25.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:25.409+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:25.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:25.542+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:25.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:25.542+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:26.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:26.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:26.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:26.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:27.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:27.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:27.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:27.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:28.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:28.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:28.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:28.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:29.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.543+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:29.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.543+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:29.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:29.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:29.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15191 cs=0 l=1 rev1=1 rx=0x7f63f0027290 tx=0x7f63f007bfd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:29.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15191 cs=0 l=1 rev1=1 rx=0x7f63f0027290 tx=0x7f63f007bfd0).stop 2021-03-09T10:20:29.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:29.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:29.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:29.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:20:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:29.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:29.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:29.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:29.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:29.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:20:29.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:29.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:20:29.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:29.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:20:29.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:29.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:29.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.654+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:29.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.655+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:29.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.655+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:29.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.655+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:29.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.655+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:20:29.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.655+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:29.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.655+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:29.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.656+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:29.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.656+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:29.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.656+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:29.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:29.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:29.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:29.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:29.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:29.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:20:29.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:20:29.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:29.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:29.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006de90 con 0x7f63d8003840 2021-03-09T10:20:29.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:29.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:29.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e480 con 0x7f63d8003840 2021-03-09T10:20:29.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:20:29.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:29.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a5a0 secure :-1 s=READY pgs=15211 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:29.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:20:29.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:29.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-03-09T10:20:29.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2021-03-09T10:20:29.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a5a0 secure :-1 s=READY pgs=15211 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f004db20).write_message error sending 0x7f63e0000c70, (32) Broken pipe 2021-03-09T10:20:29.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a5a0 secure :-1 s=READY pgs=15211 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f004db20).write_event send msg failed 2021-03-09T10:20:29.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a5a0 secure :-1 s=READY pgs=15211 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f004db20).write_event send msg failed 2021-03-09T10:20:29.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a5a0 secure :-1 s=READY pgs=15211 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f004db20).stop 2021-03-09T10:20:29.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:29.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:29.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a5a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:29.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a5a0 unknown :-1 s=CLOSED pgs=15211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:29.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:20:29.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:29.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:29.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:29.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:29.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:29.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:29.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:29.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T10:20:29.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:29.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021540 con 0x7f63d8009130 2021-03-09T10:20:29.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:29.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63e0018390 2021-03-09T10:20:29.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:29.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:29.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:29.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:29.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:29.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:29.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:20:29.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:29.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:29.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:29.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:29.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.658+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:29.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:29.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:29.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:29.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:29.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:29.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:20:29.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:20:29.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:29.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:29.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:20:29.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:29.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:29.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006de90 con 0x7f63e0018390 2021-03-09T10:20:29.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:20:29.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:29.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:20:29.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15214 cs=0 l=1 rev1=1 rx=0x7f63f007c040 tx=0x7f63f0041f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:29.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.659+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:29.683 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:29.660+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42094 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:29.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00575a0 con 0x7f63e0018390 2021-03-09T10:20:29.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.661+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:29.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.661+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:29.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.661+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:29.684 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:29.685 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:29.685 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:29.685 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:29.685 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:29.685 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:29.686 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:29.686 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:29.686 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:29.686 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:29.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.666+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:20:29.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.666+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:29.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.666+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:20:29.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:29.667+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0057390 con 0x7f63e0018390 2021-03-09T10:20:30.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:30.410+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:30.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:30.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:30.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:30.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:30.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:30.544+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:31.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:31.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:31.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:31.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:32.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:32.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:32.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:32.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:33.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:33.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:33.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:33.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:34.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:34.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:34.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:34.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:34.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15214 cs=0 l=1 rev1=1 rx=0x7f63f007c040 tx=0x7f63f0041f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:34.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15214 cs=0 l=1 rev1=1 rx=0x7f63f007c040 tx=0x7f63f0041f00).stop 2021-03-09T10:20:34.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:34.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:34.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:34.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:34.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:20:34.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:34.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:34.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:34.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:34.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:34.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:34.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:34.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:20:34.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:34.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:20:34.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:34.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00209d0 con 0x7f63d8003840 2021-03-09T10:20:34.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:34.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:34.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.661+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:34.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.662+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:34.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.662+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:34.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.662+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:34.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.662+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:20:34.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.662+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:34.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.662+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:34.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.662+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:34.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.662+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:34.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.662+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:34.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:34.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:34.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:34.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:34.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:34.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:20:34.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:20:34.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:34.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:34.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000e480 con 0x7f63d8003840 2021-03-09T10:20:34.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:34.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:34.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:20:34.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:20:34.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.663+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:34.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ac20 secure :-1 s=READY pgs=15220 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f003ed70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:34.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.664+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:20:34.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.664+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:34.677 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:34.667+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42132 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:34.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003e390 con 0x7f63d8003840 2021-03-09T10:20:34.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.668+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:34.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.668+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:34.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.668+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:34.678 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:34.679 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:34.679 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:34.679 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:34.680 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:34.680 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:34.680 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:34.681 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:34.681 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:34.681 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:34.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.669+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:20:34.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.669+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:34.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.669+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:20:34.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:34.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0057390 con 0x7f63d8003840 2021-03-09T10:20:35.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:35.411+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:35.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:35.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:35.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:35.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:36.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:36.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:36.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:36.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:37.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:37.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:37.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:37.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:38.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:38.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:38.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:38.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:39.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:39.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:39.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:20:39.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:39.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ac20 secure :-1 s=READY pgs=15220 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f003ed70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:39.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ac20 secure :-1 s=READY pgs=15220 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f003ed70).stop 2021-03-09T10:20:39.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:39.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:39.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ac20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:39.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ac20 unknown :-1 s=CLOSED pgs=15220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:39.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:20:39.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:39.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:39.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:39.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:39.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:39.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:39.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:39.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:20:39.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:39.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b600 con 0x7f63d8009130 2021-03-09T10:20:39.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:39.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63e0018390 2021-03-09T10:20:39.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:39.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:39.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:39.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:20:39.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.668+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:39.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:39.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:39.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:39.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:20:39.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:20:39.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:39.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:39.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0008ec0 con 0x7f63e0018390 2021-03-09T10:20:39.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:39.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:39.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000e480 con 0x7f63e0018390 2021-03-09T10:20:39.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:20:39.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.669+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:39.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.670+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:20:39.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.670+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:39.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.670+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15233 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f003e390).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:39.684 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:39.674+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42172 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.674+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028e30 con 0x7f63e0018390 2021-03-09T10:20:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.674+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.674+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.674+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:39.686 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:39.687 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:39.687 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:39.687 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:39.687 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:39.687 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:39.688 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:39.688 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:39.688 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:39.688 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:39.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.675+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0029c90 con 0x7f63e0018390 2021-03-09T10:20:39.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.675+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:39.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.675+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:20:39.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:39.676+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00603b0 con 0x7f63e0018390 2021-03-09T10:20:40.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:40.412+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:40.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:40.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:40.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:40.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:40.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:40.545+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:41.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:41.544+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:41.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:41.544+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:42.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:42.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:42.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:42.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:43.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:43.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:43.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:43.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.545+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:44.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.545+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:44.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:44.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:44.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15233 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f003e390).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:44.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15233 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f003e390).stop 2021-03-09T10:20:44.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:20:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:44.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:20:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:44.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017ca0 con 0x7f63d8009130 2021-03-09T10:20:44.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:44.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8003840 2021-03-09T10:20:44.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:44.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:44.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:44.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:44.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:44.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:44.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:20:44.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:44.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:44.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:44.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.675+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:44.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:44.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:44.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:20:44.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:20:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027c30 con 0x7f63d8003840 2021-03-09T10:20:44.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:44.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:44.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013210 con 0x7f63d8003840 2021-03-09T10:20:44.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:20:44.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.676+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:44.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.677+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:20:44.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.677+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 secure :-1 s=READY pgs=15248 cs=0 l=1 rev1=1 rx=0x7f63e001c010 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:44.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.677+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:44.690 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:44.680+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42208 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:44.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.681+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f680 con 0x7f63d8003840 2021-03-09T10:20:44.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.681+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:44.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.681+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:44.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.681+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:44.691 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:44.692 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:44.692 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:44.692 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:44.692 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:44.692 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:44.693 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:44.693 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:44.693 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:44.693 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:44.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.684+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f680 con 0x7f63d8003840 2021-03-09T10:20:44.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.684+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:44.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.684+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:20:44.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:44.684+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:20:45.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:45.413+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:45.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:45.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:45.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:45.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:46.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:46.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:47.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:47.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:47.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:47.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:48.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:48.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:48.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:48.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:49.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:49.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:49.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.681+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:49.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.681+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:49.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 secure :-1 s=READY pgs=15248 cs=0 l=1 rev1=1 rx=0x7f63e001c010 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:49.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 secure :-1 s=READY pgs=15248 cs=0 l=1 rev1=1 rx=0x7f63e001c010 tx=0x7f63fc04a6d0).stop 2021-03-09T10:20:49.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:49.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:49.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:49.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 unknown :-1 s=CLOSED pgs=15248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:49.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:20:49.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:49.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:49.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:49.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:49.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:49.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:49.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:49.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:20:49.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:49.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068570 con 0x7f63d8009130 2021-03-09T10:20:49.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:49.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017ca0 con 0x7f63e0018390 2021-03-09T10:20:49.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:49.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:49.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:49.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:49.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:49.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:49.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:20:49.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:49.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:49.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:49.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:49.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.682+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:49.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:49.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:49.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:49.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:49.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:49.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:20:49.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:20:49.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:49.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:49.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f63e0018390 2021-03-09T10:20:49.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:49.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:49.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027c30 con 0x7f63e0018390 2021-03-09T10:20:49.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:20:49.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:49.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.683+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15257 cs=0 l=1 rev1=1 rx=0x7f63fc00b700 tx=0x7f63fc078720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:49.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.684+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:20:49.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.684+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:49.697 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:49.687+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42250 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:49.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63e0018390 2021-03-09T10:20:49.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.688+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:49.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.688+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:49.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.688+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:49.699 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:49.699 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:49.699 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:49.699 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:49.700 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:49.700 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:49.700 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:49.700 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:49.700 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:49.701 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:49.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058000 con 0x7f63e0018390 2021-03-09T10:20:49.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.688+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:49.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.688+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:20:49.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:49.689+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63e0018390 2021-03-09T10:20:50.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:50.414+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:50.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:50.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:50.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:50.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:50.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:50.546+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:51.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:51.546+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:51.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:51.546+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:52.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:52.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:52.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:52.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:53.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:53.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:53.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:53.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:54.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:54.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:54.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:54.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:54.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15257 cs=0 l=1 rev1=1 rx=0x7f63fc00b700 tx=0x7f63fc078720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:54.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15257 cs=0 l=1 rev1=1 rx=0x7f63fc00b700 tx=0x7f63fc078720).stop 2021-03-09T10:20:54.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:54.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:54.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:54.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:54.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:20:54.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:54.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:54.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:54.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:54.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:54.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:54.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:54.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028e00 con 0x7f63d8003400 2021-03-09T10:20:54.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:54.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:20:54.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:54.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e920 con 0x7f63d8003840 2021-03-09T10:20:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:54.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.689+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.689+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:20:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.689+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:54.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.689+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.689+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:54.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.689+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:54.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.689+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:54.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:54.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:54.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:54.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:54.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:54.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:20:54.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:20:54.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:54.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:20:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:54.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f63d8003840 2021-03-09T10:20:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:20:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.690+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.691+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:20:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.691+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.691+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ba90 secure :-1 s=READY pgs=15278 cs=0 l=1 rev1=1 rx=0x7f63e002ebd0 tx=0x7f63fc01b160).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:54.703 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:54.694+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42288 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:54.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f8e0 con 0x7f63d8003840 2021-03-09T10:20:54.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.694+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:54.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.694+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:54.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.694+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:54.704 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:54.705 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:54.705 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:54.705 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:54.705 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:54.705 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:54.706 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:54.706 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:54.706 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:54.706 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:54.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.698+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f8e0 con 0x7f63d8003840 2021-03-09T10:20:54.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.698+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:54.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.699+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:20:54.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:54.699+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f550 con 0x7f63d8003840 2021-03-09T10:20:55.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:55.415+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:20:55.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:55.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:55.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:55.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:56.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:56.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:56.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:56.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:57.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:57.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:57.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:57.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:58.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:58.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:58.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:58.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:59.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.547+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:20:59.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.547+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:59.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:20:59.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:20:59.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ba90 secure :-1 s=READY pgs=15278 cs=0 l=1 rev1=1 rx=0x7f63e002ebd0 tx=0x7f63fc01b160).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:20:59.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ba90 secure :-1 s=READY pgs=15278 cs=0 l=1 rev1=1 rx=0x7f63e002ebd0 tx=0x7f63fc01b160).stop 2021-03-09T10:20:59.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:59.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:20:59.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000ba90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:20:59.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ba90 unknown :-1 s=CLOSED pgs=15278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:20:59.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:20:59.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:59.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:20:59.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:59.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:20:59.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:20:59.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:20:59.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:59.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00241f0 con 0x7f63d8003400 2021-03-09T10:20:59.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:59.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b000 con 0x7f63d8009130 2021-03-09T10:20:59.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:20:59.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017ca0 con 0x7f63e0018390 2021-03-09T10:20:59.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:20:59.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:20:59.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:20:59.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:59.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:20:59.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.695+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:20:59.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.695+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:20:59.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.695+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:20:59.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.695+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:20:59.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.695+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:20:59.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.695+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:20:59.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.695+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:20:59.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:20:59.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:59.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:59.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:20:59.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:20:59.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:20:59.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:20:59.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:20:59.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:59.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00092d0 con 0x7f63e0018390 2021-03-09T10:20:59.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:20:59.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:20:59.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0008ec0 con 0x7f63e0018390 2021-03-09T10:20:59.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:20:59.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:20:59.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15286 cs=0 l=1 rev1=1 rx=0x7f63f00426c0 tx=0x7f63f0032440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:20:59.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:20:59.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.696+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:20:59.708 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:20:59.701+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42328 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:20:59.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.701+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63e0018390 2021-03-09T10:20:59.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.701+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:20:59.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.701+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:20:59.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.701+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:20:59.710 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:20:59.710 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:20:59.710 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:59.710 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:20:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:20:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:20:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:20:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:20:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:20:59.712 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:20:59.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.701+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T10:20:59.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.701+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:20:59.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.701+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:20:59.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:20:59.703+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:21:00.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:00.416+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:00.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:00.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:00.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:00.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:00.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:00.548+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:01.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:01.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:01.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:01.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:02.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:02.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:02.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:02.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:03.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:03.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:03.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:03.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:04.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:04.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:04.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15286 cs=0 l=1 rev1=1 rx=0x7f63f00426c0 tx=0x7f63f0032440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15286 cs=0 l=1 rev1=1 rx=0x7f63f00426c0 tx=0x7f63f0032440).stop 2021-03-09T10:21:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:21:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:04.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:04.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:04.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:04.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:04.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015ab0 con 0x7f63d8003400 2021-03-09T10:21:04.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:04.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:21:04.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:04.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb30 con 0x7f63d8003840 2021-03-09T10:21:04.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.701+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:04.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.702+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:04.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.702+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:04.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.702+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:04.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.702+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:04.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.702+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:04.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.702+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:21:04.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.702+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:04.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.702+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:04.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:04.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:04.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:04.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:04.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:04.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:04.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:04.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:04.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:04.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:04.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:04.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:04.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013210 con 0x7f63d8003840 2021-03-09T10:21:04.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:04.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:04.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00092d0 con 0x7f63d8003840 2021-03-09T10:21:04.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:04.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.703+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:04.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.704+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c2e0 secure :-1 s=READY pgs=15303 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:04.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.704+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:21:04.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.704+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:04.716 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:04.707+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42366 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:04.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036d90 con 0x7f63d8003840 2021-03-09T10:21:04.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.708+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:04.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.708+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:04.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.708+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:04.718 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:04.718 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:04.718 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:04.718 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:04.718 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:04.719 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:04.719 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:04.719 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:04.719 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:04.720 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:04.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.711+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0036d90 con 0x7f63d8003840 2021-03-09T10:21:04.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.711+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:04.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.711+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:21:04.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:04.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8003840 2021-03-09T10:21:05.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:05.417+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:05.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:05.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:05.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:05.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:06.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:06.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:06.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:06.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:07.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:07.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:07.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:07.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:08.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:08.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:08.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:08.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:09.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:09.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:09.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c2e0 secure :-1 s=READY pgs=15303 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c2e0 secure :-1 s=READY pgs=15303 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f0000c30).stop 2021-03-09T10:21:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c2e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:09.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c2e0 unknown :-1 s=CLOSED pgs=15303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:09.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:21:09.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:09.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:09.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:09.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:09.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:09.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:09.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:09.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8003400 2021-03-09T10:21:09.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:09.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017ca0 con 0x7f63d8009130 2021-03-09T10:21:09.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:09.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:21:09.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:09.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:09.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:09.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:09.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:09.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.708+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:09.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.709+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:21:09.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.709+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:09.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.709+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:09.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.709+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:09.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.709+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:09.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.709+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:09.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:09.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:09.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:09.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:09.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:09.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:09.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:09.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:09.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:09.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00679c0 con 0x7f63e0018390 2021-03-09T10:21:09.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:09.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:09.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013210 con 0x7f63e0018390 2021-03-09T10:21:09.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:09.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:09.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15324 cs=0 l=1 rev1=1 rx=0x7f63f0066fe0 tx=0x7f63f0036e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:09.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:21:09.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.712+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:09.723 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:09.714+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42404 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.715+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00556a0 con 0x7f63e0018390 2021-03-09T10:21:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.715+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.715+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:09.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.715+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:09.725 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:09.726 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:09.726 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:09.726 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:09.726 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:09.727 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63e0018390 2021-03-09T10:21:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.720+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.720+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:21:09.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:09.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055b60 con 0x7f63e0018390 2021-03-09T10:21:10.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:10.418+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:10.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:10.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:10.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:10.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:10.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:10.549+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:11.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:11.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:11.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:11.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:12.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:12.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:12.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:12.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:13.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:13.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:13.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:13.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:14.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:14.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:14.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:14.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:14.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15324 cs=0 l=1 rev1=1 rx=0x7f63f0066fe0 tx=0x7f63f0036e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:14.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15324 cs=0 l=1 rev1=1 rx=0x7f63f0066fe0 tx=0x7f63f0036e70).stop 2021-03-09T10:21:14.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:14.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:14.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:14.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:14.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:21:14.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:14.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:14.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:14.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:14.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:14.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:14.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:14.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b000 con 0x7f63d8003400 2021-03-09T10:21:14.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:14.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cda0 con 0x7f63d8009130 2021-03-09T10:21:14.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:14.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:21:14.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:14.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:14.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:14.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:14.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:14.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:14.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.715+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:21:14.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.715+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:14.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.715+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:14.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.715+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:14.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.715+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:14.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.715+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:14.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:14.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:14.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:14.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:14.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:14.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:14.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:14.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:14.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:14.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0023050 con 0x7f63d8003840 2021-03-09T10:21:14.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:14.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:14.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00679c0 con 0x7f63d8003840 2021-03-09T10:21:14.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:14.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:14.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 secure :-1 s=READY pgs=15332 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0042400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:14.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:21:14.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.716+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:14.729 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:14.719+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42444 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:14.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e430 con 0x7f63d8003840 2021-03-09T10:21:14.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.720+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:14.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.720+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:14.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.720+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:14.732 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:14.732 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:14.732 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:14.733 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:14.733 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:14.733 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:14.733 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:14.734 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:14.734 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:14.734 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:14.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.723+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:21:14.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.723+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:14.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.723+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:21:14.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:14.723+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036a20 con 0x7f63d8003840 2021-03-09T10:21:15.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:15.418+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:15.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:15.548+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:15.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:15.548+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:16.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:16.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:16.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:16.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:17.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:17.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:17.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:17.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:18.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:18.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:18.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:18.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:19.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:19.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:19.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:19.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:19.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 secure :-1 s=READY pgs=15332 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0042400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:19.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 secure :-1 s=READY pgs=15332 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0042400).stop 2021-03-09T10:21:19.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:19.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:19.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:19.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 unknown :-1 s=CLOSED pgs=15332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:19.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:21:19.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:19.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:19.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:19.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:19.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:19.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:19.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:19.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025c60 con 0x7f63d8003400 2021-03-09T10:21:19.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:19.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:21:19.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:19.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63e0018390 2021-03-09T10:21:19.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:19.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:19.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:19.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:19.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:21:19.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:19.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:19.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:19.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:19.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:19.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:19.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.720+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:19.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:19.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:19.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:19.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:19.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:19.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:19.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:19.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:19.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:19.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024b00 con 0x7f63e0018390 2021-03-09T10:21:19.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:19.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:19.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0023050 con 0x7f63e0018390 2021-03-09T10:21:19.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:19.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.721+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:19.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.722+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15342 cs=0 l=1 rev1=1 rx=0x7f63e007ab80 tx=0x7f63f003ec80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:19.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.722+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:21:19.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.722+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:19.735 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:19.727+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42484 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:19.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005fe80 con 0x7f63e0018390 2021-03-09T10:21:19.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.728+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:19.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.728+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:19.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.728+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:19.736 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:19.737 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:19.737 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:19.737 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:19.737 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:19.737 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:19.738 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:19.738 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:19.738 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:19.738 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:19.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:21:19.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.728+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:19.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.728+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:21:19.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:19.729+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0039330 con 0x7f63e0018390 2021-03-09T10:21:20.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:20.419+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:20.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:20.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:20.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:20.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:20.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:20.549+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:21.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:21.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:21.549+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:22.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:22.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:22.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:22.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:23.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:23.549+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:23.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:23.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:24.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:24.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:24.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:24.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:24.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15342 cs=0 l=1 rev1=1 rx=0x7f63e007ab80 tx=0x7f63f003ec80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:24.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15342 cs=0 l=1 rev1=1 rx=0x7f63e007ab80 tx=0x7f63f003ec80).stop 2021-03-09T10:21:24.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:24.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:24.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:24.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:24.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:21:24.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:24.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:24.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:24.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:24.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:24.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:24.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:24.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017ca0 con 0x7f63d8003400 2021-03-09T10:21:24.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:24.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019a90 con 0x7f63d8009130 2021-03-09T10:21:24.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:24.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:21:24.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:24.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:24.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:24.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:24.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:21:24.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:24.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:24.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:24.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:24.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.728+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:24.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:24.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:24.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:24.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:24.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:24.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:24.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:24.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:21:24.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:21:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a710 con 0x7f63d8003840 2021-03-09T10:21:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:24.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:24.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:21:24.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:21:24.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:24.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:21:24.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 secure :-1 s=READY pgs=15356 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc06f8a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:24.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.729+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:24.743 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:24.734+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42522 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:24.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.734+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f63d8003840 2021-03-09T10:21:24.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.734+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:24.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.734+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:24.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.734+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:24.745 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:24.745 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:24.745 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:24.746 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:24.746 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:24.746 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:24.746 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:24.747 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:24.747 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:24.747 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:24.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.734+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f63d8003840 2021-03-09T10:21:24.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.734+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:24.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.734+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:21:24.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:24.736+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:21:25.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:25.420+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:25.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:25.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:25.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:25.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:26.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:26.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:26.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:26.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:27.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:27.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:27.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:27.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:28.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:28.550+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:28.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:28.550+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:29.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:29.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:29.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.734+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:29.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.734+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:29.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.734+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 secure :-1 s=READY pgs=15356 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc06f8a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:29.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 secure :-1 s=READY pgs=15356 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc06f8a0).stop 2021-03-09T10:21:29.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:29.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:29.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:29.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079460 unknown :-1 s=CLOSED pgs=15356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:29.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:21:29.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:29.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:29.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:29.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:29.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:29.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:29.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:29.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002cd0 con 0x7f63d8003400 2021-03-09T10:21:29.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:29.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:21:29.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:29.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63e0018390 2021-03-09T10:21:29.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:29.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:29.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:29.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:29.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.735+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:29.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.736+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:29.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.736+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:21:29.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.736+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:29.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.736+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:29.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.736+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:29.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.736+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:29.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.736+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:29.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:29.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:29.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:29.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:29.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:29.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:29.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:29.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:29.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:29.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001030 con 0x7f63e0018390 2021-03-09T10:21:29.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:29.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:29.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024b00 con 0x7f63e0018390 2021-03-09T10:21:29.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:29.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.737+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:29.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.738+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15375 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f002d290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:29.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.738+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:21:29.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.738+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:29.750 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:29.742+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42562 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:29.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:21:29.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.743+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:29.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.743+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:29.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.743+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:29.752 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:29.752 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:29.752 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:29.753 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:29.753 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:29.753 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:29.753 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:29.754 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:29.754 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:29.754 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:29.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.746+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:21:29.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.746+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:29.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.746+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:21:29.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:29.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00284b0 con 0x7f63e0018390 2021-03-09T10:21:30.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:30.421+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:30.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:30.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:30.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:30.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:30.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:30.551+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:31.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:31.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:31.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:31.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:32.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:32.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:32.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:33.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:33.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:33.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:33.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:34.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:34.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:34.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:34.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:34.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15375 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f002d290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:34.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15375 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f002d290).stop 2021-03-09T10:21:34.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:34.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:34.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:34.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:34.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:21:34.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:34.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:34.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:34.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:34.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:34.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:34.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:34.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b140 con 0x7f63d8003400 2021-03-09T10:21:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018ed0 con 0x7f63d8009130 2021-03-09T10:21:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019a90 con 0x7f63d8003840 2021-03-09T10:21:34.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:34.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:21:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:34.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:34.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.743+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:34.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:34.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:34.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:34.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:34.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:34.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:34.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:34.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024b00 con 0x7f63d8003840 2021-03-09T10:21:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:34.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001030 con 0x7f63d8003840 2021-03-09T10:21:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079b60 secure :-1 s=READY pgs=15382 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004c2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:21:34.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.744+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:34.758 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:34.750+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42600 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.750+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027e70 con 0x7f63d8003840 2021-03-09T10:21:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.750+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.750+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:34.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.750+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:34.760 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:34.760 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:34.760 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:34.760 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:34.760 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:34.761 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:34.761 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:34.761 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:34.761 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:34.762 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:34.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.750+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:21:34.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.750+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:34.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.750+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:21:34.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:34.752+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0031900 con 0x7f63d8003840 2021-03-09T10:21:35.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:35.422+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:35.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:35.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:35.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:35.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:36.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:36.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:36.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:36.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:37.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:37.551+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:37.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:37.551+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:38.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:38.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:38.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:38.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:39.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:39.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:39.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:39.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:39.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079b60 secure :-1 s=READY pgs=15382 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004c2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:39.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079b60 secure :-1 s=READY pgs=15382 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f004c2d0).stop 2021-03-09T10:21:39.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:39.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:39.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079b60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:39.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079b60 unknown :-1 s=CLOSED pgs=15382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:39.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:21:39.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:39.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.750+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:39.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:39.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:39.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:39.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:39.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:39.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266d0 con 0x7f63d8003400 2021-03-09T10:21:39.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:39.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc60 con 0x7f63d8009130 2021-03-09T10:21:39.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:39.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e0018390 2021-03-09T10:21:39.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:39.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:39.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:39.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:39.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:39.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.751+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:39.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:21:39.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:39.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:39.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:39.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:39.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:39.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:39.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:39.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:39.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:39.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:39.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:39.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:39.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:39.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:39.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0008ee0 con 0x7f63e0018390 2021-03-09T10:21:39.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:39.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:39.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024b00 con 0x7f63e0018390 2021-03-09T10:21:39.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:39.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.752+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:39.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.754+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15400 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f00669e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:39.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.754+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:21:39.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.754+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:39.766 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:39.757+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42638 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:39.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:21:39.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.758+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:39.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.758+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:39.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.758+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:39.768 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:39.768 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:39.768 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:39.768 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:39.769 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:39.769 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:39.769 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:39.769 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:39.769 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:39.770 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.759+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:21:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.759+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.759+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:21:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:39.760+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:21:40.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:40.423+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:40.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:40.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:40.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:40.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:40.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:40.552+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:41.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:41.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:41.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:41.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:42.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:42.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:42.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:42.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:43.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:43.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:43.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:43.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:44.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.552+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:44.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.552+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:44.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:21:44.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:44.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15400 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f00669e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:44.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15400 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f00669e0).stop 2021-03-09T10:21:44.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:44.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:44.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:44.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:44.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:21:44.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:44.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:44.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:44.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:44.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:44.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:44.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:44.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:21:44.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:44.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cfe0 con 0x7f63d8009130 2021-03-09T10:21:44.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:44.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:21:44.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:44.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:44.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:44.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:44.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.758+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:44.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:44.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:21:44.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:44.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:44.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:44.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:44.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:44.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:44.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:44.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:44.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:44.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:44.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:44.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:44.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:44.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:44.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024b00 con 0x7f63d8003840 2021-03-09T10:21:44.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:44.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:44.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0008ee0 con 0x7f63d8003840 2021-03-09T10:21:44.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:44.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.759+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:44.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.760+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e9c0 secure :-1 s=READY pgs=15411 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007e330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:44.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.760+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:21:44.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.760+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:44.773 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:44.764+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42678 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:44.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.765+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027e70 con 0x7f63d8003840 2021-03-09T10:21:44.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.765+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:44.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.765+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:44.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.765+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:44.774 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:44.774 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:44.774 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:44.775 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:44.775 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:44.775 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:44.776 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:44.776 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:44.776 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:44.776 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:44.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.769+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:21:44.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.769+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:44.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.769+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:21:44.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:44.769+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:21:45.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:45.424+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:45.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:45.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:45.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:45.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:46.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:46.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:46.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:46.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:47.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:47.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:47.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:47.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:48.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:48.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:48.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:48.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:49.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:49.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:49.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:50.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:50.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e9c0 secure :-1 s=READY pgs=15411 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007e330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:50.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e9c0 secure :-1 s=READY pgs=15411 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007e330).stop 2021-03-09T10:21:50.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:50.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:50.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e9c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:50.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e9c0 unknown :-1 s=CLOSED pgs=15411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:50.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:21:50.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:50.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:50.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:50.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:50.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:50.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:50.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:50.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:21:50.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:50.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266d0 con 0x7f63d8009130 2021-03-09T10:21:50.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:50.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb30 con 0x7f63e0018390 2021-03-09T10:21:50.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:50.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:50.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:50.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.765+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:50.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.765+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:50.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.765+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:50.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.765+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:21:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.765+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.765+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.765+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:50.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.765+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.766+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:50.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:50.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:50.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:50.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:50.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:50.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0064f50 con 0x7f63e0018390 2021-03-09T10:21:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:50.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:50.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024b00 con 0x7f63e0018390 2021-03-09T10:21:50.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:50.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.767+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:50.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.768+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15432 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f00603b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:50.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.768+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:21:50.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.768+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:50.078 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:49.770+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42717 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:50.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0060580 con 0x7f63e0018390 2021-03-09T10:21:50.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.771+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:50.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.771+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:50.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.771+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:50.080 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:50.080 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:50.081 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:50.081 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:50.081 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:50.081 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:50.082 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:50.082 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:50.082 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:50.083 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:50.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0038c30 con 0x7f63e0018390 2021-03-09T10:21:50.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.776+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:50.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0066af0 con 0x7f63e0018390 2021-03-09T10:21:50.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:49.776+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:21:50.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:50.425+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:50.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:50.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:50.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:50.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:50.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:50.553+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:51.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:51.553+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:51.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:51.553+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:52.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:52.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:52.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:53.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:53.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:53.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:53.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:54.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:54.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:54.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:54.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:54.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15432 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f00603b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:54.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15432 cs=0 l=1 rev1=1 rx=0x7f63f000dfb0 tx=0x7f63f00603b0).stop 2021-03-09T10:21:54.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:54.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:54.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:54.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:54.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:21:54.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00116b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:54.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:54.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:54.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:54.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:54.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:54.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:54.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:21:54.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:54.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:21:54.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:54.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003840 2021-03-09T10:21:54.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:54.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:54.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:54.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:54.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.771+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:54.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.772+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00116b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:54.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.772+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:21:54.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.772+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:54.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.772+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:54.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.772+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:54.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.772+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:54.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.772+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:54.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:54.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:54.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:54.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:54.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:54.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:21:54.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:21:54.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:54.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:54.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c160 con 0x7f63d8003840 2021-03-09T10:21:54.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:54.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:54.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.773+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0064f50 con 0x7f63d8003840 2021-03-09T10:21:54.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.774+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:21:54.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.774+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:54.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.775+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:21:54.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.775+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00116b0 secure :-1 s=READY pgs=15448 cs=0 l=1 rev1=1 rx=0x7f63f0040800 tx=0x7f63f0039fc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:54.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.775+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:54.787 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:54.777+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42756 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:54.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.778+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0031500 con 0x7f63d8003840 2021-03-09T10:21:54.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.778+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:54.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.778+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:54.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.778+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:54.788 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:54.788 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:54.789 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:54.789 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:54.789 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:54.789 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:54.789 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:54.790 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:54.790 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:54.790 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:54.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:21:54.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.782+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:54.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.782+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8003840 2021-03-09T10:21:54.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:54.782+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:21:55.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:55.426+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:21:55.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:55.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:55.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:55.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:56.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:56.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:56.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:56.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:57.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:57.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:57.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:57.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:58.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:58.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:58.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:58.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:59.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.554+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:21:59.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.554+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:59.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00116b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:21:59.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00116b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:21:59.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00116b0 secure :-1 s=READY pgs=15448 cs=0 l=1 rev1=1 rx=0x7f63f0040800 tx=0x7f63f0039fc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:21:59.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00116b0 secure :-1 s=READY pgs=15448 cs=0 l=1 rev1=1 rx=0x7f63f0040800 tx=0x7f63f0039fc0).stop 2021-03-09T10:21:59.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:59.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:21:59.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00116b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:21:59.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00116b0 unknown :-1 s=CLOSED pgs=15448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:21:59.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:21:59.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:59.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:21:59.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:59.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:21:59.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:21:59.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:21:59.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:59.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:21:59.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:59.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:21:59.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:21:59.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017ca0 con 0x7f63e0018390 2021-03-09T10:21:59.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:21:59.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:21:59.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.780+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:59.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.781+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:21:59.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.781+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:21:59.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.781+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:21:59.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.781+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:21:59.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.781+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:21:59.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.781+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:21:59.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.781+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:21:59.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.781+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:21:59.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.781+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:21:59.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:21:59.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e002dc30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:59.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e002dc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:59.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:21:59.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:21:59.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:21:59.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:21:59.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:21:59.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:59.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024b00 con 0x7f63e0018390 2021-03-09T10:21:59.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:21:59.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:21:59.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a710 con 0x7f63e0018390 2021-03-09T10:21:59.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:21:59.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:21:59.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:21:59.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:21:59.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.782+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15461 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc03ff80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:21:59.796 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:21:59.785+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42794 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:21:59.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:21:59.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.787+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:21:59.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.787+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:21:59.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.787+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:21:59.798 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:21:59.798 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:21:59.798 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:59.799 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:21:59.799 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:21:59.799 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:21:59.799 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:21:59.799 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:21:59.800 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:21:59.800 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:21:59.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.790+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63e0018390 2021-03-09T10:21:59.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.790+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:21:59.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.790+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:21:59.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:21:59.790+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:22:00.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:00.427+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:00.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:00.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:00.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:00.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:00.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:00.555+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:01.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:01.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:01.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:01.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:02.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:02.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:02.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:02.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:03.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:03.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:03.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:03.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:04.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:04.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:04.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:04.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:04.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15461 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc03ff80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:04.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15461 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc03ff80).stop 2021-03-09T10:22:04.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:04.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:04.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:04.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:04.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:22:04.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:04.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:04.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:04.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:04.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:04.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:04.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:04.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b920 con 0x7f63d8003400 2021-03-09T10:22:04.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:04.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022580 con 0x7f63d8009130 2021-03-09T10:22:04.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:04.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:22:04.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:04.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:04.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:04.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:04.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:04.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.787+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:04.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.787+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:22:04.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.787+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:04.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.787+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:04.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.788+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:04.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.788+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:04.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.788+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:04.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:04.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:04.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:04.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:04.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:22:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:22:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:04.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:04.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b460 con 0x7f63d8003840 2021-03-09T10:22:04.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:04.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:04.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024b00 con 0x7f63d8003840 2021-03-09T10:22:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:22:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 secure :-1 s=READY pgs=15475 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc049770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:04.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:22:04.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.789+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:04.803 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:04.792+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42834 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:04.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f63d8003840 2021-03-09T10:22:04.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.792+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:04.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.792+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:04.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.792+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:04.805 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:04.805 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:04.805 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:04.806 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:04.806 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:04.806 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:04.807 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:04.807 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:04.807 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:04.807 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:04.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.794+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8003840 2021-03-09T10:22:04.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.794+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:04.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.794+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:22:04.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:04.798+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f63d8003840 2021-03-09T10:22:05.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:05.428+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:05.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:05.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:05.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:05.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:06.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:06.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:06.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:06.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:07.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:07.555+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:07.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:07.555+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:08.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:08.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:08.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:08.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:09.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:09.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:09.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:22:09.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:09.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 secure :-1 s=READY pgs=15475 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc049770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:09.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 secure :-1 s=READY pgs=15475 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc049770).stop 2021-03-09T10:22:09.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:09.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:09.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:09.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 unknown :-1 s=CLOSED pgs=15475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:09.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:22:09.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:09.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:09.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015950 con 0x7f63d8003400 2021-03-09T10:22:09.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e4d0 con 0x7f63d8009130 2021-03-09T10:22:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:09.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014620 con 0x7f63e0018390 2021-03-09T10:22:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:09.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.793+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:09.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.794+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:09.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.794+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:22:09.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.794+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:09.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.794+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:09.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.794+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:09.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:09.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:09.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:22:09.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:22:09.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:09.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:09.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017ca0 con 0x7f63e0018390 2021-03-09T10:22:09.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:09.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:09.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b460 con 0x7f63e0018390 2021-03-09T10:22:09.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:22:09.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.795+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:09.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.796+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:22:09.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.796+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15486 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc049a90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:09.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.796+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:09.815 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:09.800+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42872 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:09.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0787b0 con 0x7f63e0018390 2021-03-09T10:22:09.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.801+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:09.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.801+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:09.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.801+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:09.817 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:09.817 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:09.818 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:09.818 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:09.818 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:09.819 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:09.819 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:09.820 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:09.820 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:09.820 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:09.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.803+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0787b0 con 0x7f63e0018390 2021-03-09T10:22:09.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.803+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:09.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.803+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:22:09.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:09.804+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc078cb0 con 0x7f63e0018390 2021-03-09T10:22:10.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:10.429+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:10.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:10.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:10.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:10.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:10.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:10.556+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:11.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:11.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:11.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:12.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:12.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:12.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:12.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:13.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:13.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:13.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:13.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:14.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:14.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:14.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:14.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:14.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15486 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc049a90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:14.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15486 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc049a90).stop 2021-03-09T10:22:14.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:14.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:14.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:14.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:14.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:22:14.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:14.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:14.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:14.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:14.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:14.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:14.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:14.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:22:14.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:14.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bc90 con 0x7f63d8009130 2021-03-09T10:22:14.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:14.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065b90 con 0x7f63d8003840 2021-03-09T10:22:14.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:14.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:14.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.801+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:14.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:14.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:22:14.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:14.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:14.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:14.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:14.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:14.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:14.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:14.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:14.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:14.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:14.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:14.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:14.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:22:14.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:22:14.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:14.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:14.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da60 con 0x7f63d8003840 2021-03-09T10:22:14.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:14.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:14.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c160 con 0x7f63d8003840 2021-03-09T10:22:14.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:22:14.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.802+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:14.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.803+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:22:14.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.803+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:14.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.803+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 secure :-1 s=READY pgs=15496 cs=0 l=1 rev1=1 rx=0x7f63f0013010 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:14.816 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:14.808+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42912 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:14.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:22:14.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.808+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:14.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.808+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:14.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.808+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:14.817 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:14.818 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:14.818 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:14.818 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:14.818 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:14.819 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:14.819 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:14.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:22:14.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.813+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:14.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.813+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:22:14.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:14.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003ec10 con 0x7f63d8003840 2021-03-09T10:22:15.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:15.430+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:15.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:15.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:15.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:15.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:16.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:16.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:16.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:16.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:17.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:17.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:17.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:17.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:18.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:18.556+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:18.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:18.556+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:19.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:19.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:19.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:19.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:19.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 secure :-1 s=READY pgs=15496 cs=0 l=1 rev1=1 rx=0x7f63f0013010 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:19.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 secure :-1 s=READY pgs=15496 cs=0 l=1 rev1=1 rx=0x7f63f0013010 tx=0x7f63f004dd90).stop 2021-03-09T10:22:19.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:19.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:19.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:19.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0064b10 unknown :-1 s=CLOSED pgs=15496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:19.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:22:19.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:19.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:19.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:19.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:19.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:19.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:19.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:19.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a190 con 0x7f63d8003400 2021-03-09T10:22:19.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:19.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:22:19.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:19.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d920 con 0x7f63e0018390 2021-03-09T10:22:19.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:19.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:19.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.808+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:19.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.809+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:19.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.809+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:19.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.809+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:19.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.809+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:22:19.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.809+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:19.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.809+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:19.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:19.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:19.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:19.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:19.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:19.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:19.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:19.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:19.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:22:19.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:22:19.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:19.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:19.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d60 con 0x7f63e0018390 2021-03-09T10:22:19.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:19.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:19.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017ca0 con 0x7f63e0018390 2021-03-09T10:22:19.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:22:19.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.810+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:19.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.811+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:22:19.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.811+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:19.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.811+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15510 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:19.824 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:19.815+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42950 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:19.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.815+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04b040 con 0x7f63e0018390 2021-03-09T10:22:19.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.815+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:19.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.815+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:19.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.815+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:19.825 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:19.826 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:19.826 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:19.826 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:19.826 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:19.827 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:19.827 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04b040 con 0x7f63e0018390 2021-03-09T10:22:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.816+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.816+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:22:19.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:19.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0018390 2021-03-09T10:22:20.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:20.431+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:20.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:20.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:20.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:20.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:20.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:20.557+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:21.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:21.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:21.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:21.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:22.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:22.557+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:22.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:22.557+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:23.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:23.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:23.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:23.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:24.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:24.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:24.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:24.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:24.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15510 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:24.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15510 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009990).stop 2021-03-09T10:22:24.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:24.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:24.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:24.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:24.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:22:24.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:24.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:24.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00651c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:24.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:24.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:24.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:24.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:24.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:22:24.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:24.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028040 con 0x7f63d8009130 2021-03-09T10:22:24.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:24.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b140 con 0x7f63d8003840 2021-03-09T10:22:24.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:24.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:24.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:24.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:24.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:24.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00651c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:24.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:22:24.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:24.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:24.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:24.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:24.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:24.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:24.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:24.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:24.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:24.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:24.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:22:24.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:22:24.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:24.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:24.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00198e0 con 0x7f63d8003840 2021-03-09T10:22:24.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:24.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:24.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d60 con 0x7f63d8003840 2021-03-09T10:22:24.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:22:24.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.816+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:24.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.818+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00651c0 secure :-1 s=READY pgs=15524 cs=0 l=1 rev1=1 rx=0x7f63e006b620 tx=0x7f63fc04b4c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:24.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.818+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:22:24.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.818+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:24.830 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:24.822+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 42990 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:24.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc05de90 con 0x7f63d8003840 2021-03-09T10:22:24.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.823+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:24.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.823+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:24.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.823+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:24.832 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:24.832 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:24.832 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:24.832 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:24.833 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:24.833 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:24.833 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:24.833 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:24.833 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:24.834 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:24.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.827+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05de90 con 0x7f63d8003840 2021-03-09T10:22:24.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.827+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:24.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.827+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:22:24.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:24.828+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8003840 2021-03-09T10:22:25.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:25.432+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:25.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:25.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:25.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:25.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:26.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:26.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:26.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:26.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:27.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:27.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:27.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:27.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:28.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:28.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:28.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:28.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:29.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.558+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:29.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.558+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:29.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00651c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:29.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00651c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:29.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00651c0 secure :-1 s=READY pgs=15524 cs=0 l=1 rev1=1 rx=0x7f63e006b620 tx=0x7f63fc04b4c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:29.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00651c0 secure :-1 s=READY pgs=15524 cs=0 l=1 rev1=1 rx=0x7f63e006b620 tx=0x7f63fc04b4c0).stop 2021-03-09T10:22:29.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:29.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:29.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00651c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:29.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00651c0 unknown :-1 s=CLOSED pgs=15524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:29.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:22:29.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:29.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:29.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:29.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:29.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:29.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:29.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:29.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066330 con 0x7f63d8003400 2021-03-09T10:22:29.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:29.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b920 con 0x7f63d8009130 2021-03-09T10:22:29.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:29.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015950 con 0x7f63e0018390 2021-03-09T10:22:29.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:29.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:29.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:29.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:29.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.823+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:29.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:29.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:22:29.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:29.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:29.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:29.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:29.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:29.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:29.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:29.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:29.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:29.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:29.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:22:29.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:22:29.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:29.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:29.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:22:29.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:29.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:29.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T10:22:29.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:22:29.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.824+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:29.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.825+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15536 cs=0 l=1 rev1=1 rx=0x7f63e0079a40 tx=0x7f63f007f700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:29.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.825+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:22:29.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.825+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:29.838 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:29.829+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43028 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:29.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0039df0 con 0x7f63e0018390 2021-03-09T10:22:29.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.829+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:29.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.829+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:29.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.829+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:29.841 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:29.841 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:29.841 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:29.841 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:29.841 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:29.842 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:29.842 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:29.842 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:29.842 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:29.842 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:29.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0039df0 con 0x7f63e0018390 2021-03-09T10:22:29.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.829+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:29.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.829+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:22:29.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:29.831+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:22:30.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:30.433+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:30.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:30.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:30.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:30.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:30.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:30.559+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:31.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:31.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:31.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:31.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:32.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:32.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:32.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:32.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:33.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:33.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:33.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:33.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:34.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:34.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:34.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:34.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:34.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15536 cs=0 l=1 rev1=1 rx=0x7f63e0079a40 tx=0x7f63f007f700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:34.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15536 cs=0 l=1 rev1=1 rx=0x7f63e0079a40 tx=0x7f63f007f700).stop 2021-03-09T10:22:34.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:34.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:34.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:34.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:34.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:22:34.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:34.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:34.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:34.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:34.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:34.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:34.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:34.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:22:34.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:34.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a10 con 0x7f63d8009130 2021-03-09T10:22:34.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:34.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b4c0 con 0x7f63d8003840 2021-03-09T10:22:34.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:34.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:34.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.829+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:34.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:34.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:34.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:34.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:22:34.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:34.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:34.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:34.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:34.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:34.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:34.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:34.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:34.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:34.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:34.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:22:34.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:22:34.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:34.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:34.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00266d0 con 0x7f63d8003840 2021-03-09T10:22:34.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:34.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:34.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:22:34.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:22:34.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.830+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:34.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.831+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15549 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:34.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.831+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:22:34.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.831+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:34.844 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:34.835+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43068 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:34.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.835+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f4c0 con 0x7f63d8003840 2021-03-09T10:22:34.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.835+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:34.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.835+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:34.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.835+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:34.846 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:34.846 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:34.847 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:34.847 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:34.847 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:34.847 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:34.847 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:34.848 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:34.848 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:34.848 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:34.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.835+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8003840 2021-03-09T10:22:34.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.835+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:34.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.835+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:22:34.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:34.837+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8003840 2021-03-09T10:22:35.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:35.434+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:35.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:35.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:35.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:35.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:36.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:36.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:36.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:36.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:37.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:37.559+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:37.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:37.559+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:38.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:38.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:38.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:38.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:39.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:39.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:39.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:39.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:39.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15549 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:39.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15549 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006610).stop 2021-03-09T10:22:39.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:39.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:39.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:39.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=CLOSED pgs=15549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:39.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:22:39.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:39.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:39.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:39.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:39.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:39.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:39.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:39.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:22:39.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:39.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001230 con 0x7f63d8009130 2021-03-09T10:22:39.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:39.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b920 con 0x7f63e0018390 2021-03-09T10:22:39.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:39.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:39.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:39.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:39.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:39.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:39.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:22:39.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:39.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:39.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:39.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:39.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:39.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:39.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:39.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:39.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:39.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:39.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:22:39.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:22:39.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:39.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:39.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d60 con 0x7f63e0018390 2021-03-09T10:22:39.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:39.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:39.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00266d0 con 0x7f63e0018390 2021-03-09T10:22:39.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:22:39.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.837+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:39.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.838+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15564 cs=0 l=1 rev1=1 rx=0x7f63e005e8d0 tx=0x7f63f0088e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:39.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.838+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:22:39.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.838+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:39.852 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:39.842+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43106 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:39.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63e0018390 2021-03-09T10:22:39.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.843+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:39.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.843+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:39.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.843+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:39.853 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:39.854 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:39.854 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:39.854 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:39.854 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:39.854 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:39.855 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:39.855 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:39.855 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:39.855 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:39.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f4c0 con 0x7f63e0018390 2021-03-09T10:22:39.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.846+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:39.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.846+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:22:39.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:39.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e0018390 2021-03-09T10:22:40.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:40.435+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:40.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:40.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:40.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:40.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:40.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:40.560+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:41.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:41.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:41.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:41.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:42.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:42.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:42.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:42.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:43.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:43.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:43.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:43.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:44.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.560+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:44.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.560+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:44.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:44.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:44.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15564 cs=0 l=1 rev1=1 rx=0x7f63e005e8d0 tx=0x7f63f0088e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:44.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15564 cs=0 l=1 rev1=1 rx=0x7f63e005e8d0 tx=0x7f63f0088e20).stop 2021-03-09T10:22:44.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:44.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:44.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:44.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:44.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:22:44.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:44.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:44.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:44.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:44.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:44.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:44.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:44.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b6c0 con 0x7f63d8003400 2021-03-09T10:22:44.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:44.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:22:44.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:44.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067a10 con 0x7f63d8003840 2021-03-09T10:22:44.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:44.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:44.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:44.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:44.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:44.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:44.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:22:44.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:44.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:44.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:44.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:44.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:44.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:44.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:44.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:44.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:44.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:44.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:22:44.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:22:44.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:44.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:44.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f63d8003840 2021-03-09T10:22:44.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:44.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:44.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d60 con 0x7f63d8003840 2021-03-09T10:22:44.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:22:44.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.844+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:44.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.845+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b970 secure :-1 s=READY pgs=15578 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0032f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:44.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.845+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:22:44.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.845+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:44.858 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:44.851+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43146 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:44.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:22:44.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.851+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:44.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.851+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:44.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.851+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:44.860 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:44.860 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:44.860 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:44.860 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:44.861 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:44.861 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:44.861 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:44.861 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:44.861 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:44.862 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:44.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00294d0 con 0x7f63d8003840 2021-03-09T10:22:44.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.851+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:44.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.851+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:22:44.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:44.852+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0031b70 con 0x7f63d8003840 2021-03-09T10:22:45.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:45.436+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:45.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:45.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:45.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:45.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:46.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:46.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:46.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:46.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:47.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:47.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:47.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:47.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:48.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:48.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:48.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:48.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:49.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:49.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:49.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:49.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:49.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b970 secure :-1 s=READY pgs=15578 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0032f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:49.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b970 secure :-1 s=READY pgs=15578 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0032f80).stop 2021-03-09T10:22:49.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:49.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:49.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:49.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b970 unknown :-1 s=CLOSED pgs=15578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:49.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:22:49.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:49.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:49.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:49.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:49.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:49.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:49.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:49.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b4c0 con 0x7f63d8003400 2021-03-09T10:22:49.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:49.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00668f0 con 0x7f63d8009130 2021-03-09T10:22:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb30 con 0x7f63e0018390 2021-03-09T10:22:49.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:49.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.852+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.852+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.852+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:49.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.852+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:22:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.852+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.852+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.853+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.853+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:49.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.853+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:49.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:49.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:49.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:49.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:49.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:49.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:22:49.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:22:49.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:49.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:49.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:22:49.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:49.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:49.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f63e0018390 2021-03-09T10:22:49.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:22:49.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.854+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:49.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15598 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0038cf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:49.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.855+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:22:49.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.855+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:49.866 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:49.857+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43184 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:49.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63e0018390 2021-03-09T10:22:49.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.858+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:49.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.858+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:49.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.858+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:49.867 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:49.868 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:49.868 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:49.868 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:49.868 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:49.868 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:49.869 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:49.869 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:49.869 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:49.869 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:49.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63e0018390 2021-03-09T10:22:49.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.859+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:49.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.859+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:22:49.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:49.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0031b70 con 0x7f63e0018390 2021-03-09T10:22:50.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:50.437+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:50.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:50.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:50.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:50.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:50.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:50.561+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:51.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:51.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:51.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:51.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:52.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:52.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:52.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:52.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:53.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:53.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:53.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:53.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:54.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:54.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:54.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:22:54.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:54.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15598 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0038cf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:54.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15598 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0038cf0).stop 2021-03-09T10:22:54.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:54.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:54.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:54.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:54.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:22:54.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:54.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:54.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:54.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:54.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:54.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:54.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:54.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b140 con 0x7f63d8003400 2021-03-09T10:22:54.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:54.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065b90 con 0x7f63d8009130 2021-03-09T10:22:54.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:54.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001230 con 0x7f63d8003840 2021-03-09T10:22:54.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:54.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:54.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:54.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:54.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:54.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:54.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:22:54.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:54.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.858+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:54.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:54.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:54.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:54.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:54.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:54.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:54.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:22:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:22:54.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:54.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:54.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63d8003840 2021-03-09T10:22:54.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:54.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:54.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:22:54.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:22:54.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.861+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:54.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15614 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f007f700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:54.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.862+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:22:54.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.862+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:54.873 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:54.865+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43224 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:54.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027430 con 0x7f63d8003840 2021-03-09T10:22:54.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.865+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:54.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.865+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:54.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.865+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:54.875 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:54.875 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:54.876 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:54.876 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:54.876 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:54.876 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:54.876 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:54.877 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:54.877 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:54.877 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:54.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:22:54.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.866+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.866+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:22:54.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:54.867+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:22:55.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:55.438+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:22:55.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:55.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:55.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:55.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:56.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:56.561+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:56.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:56.561+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:57.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:57.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:57.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:57.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:58.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:58.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:58.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:59.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.562+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:22:59.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.562+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:59.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:22:59.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:22:59.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15614 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f007f700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:22:59.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15614 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f007f700).stop 2021-03-09T10:22:59.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:59.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:22:59.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:22:59.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=CLOSED pgs=15614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:22:59.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:22:59.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:59.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:22:59.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:59.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:22:59.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:22:59.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:22:59.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:59.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:22:59.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:59.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:22:59.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:22:59.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e0018390 2021-03-09T10:22:59.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.865+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:22:59.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:22:59.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:59.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:22:59.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:22:59.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:22:59.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:22:59.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:22:59.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:22:59.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:22:59.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:22:59.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.866+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:22:59.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:22:59.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:59.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:59.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:22:59.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:22:59.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:22:59.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:22:59.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:22:59.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:59.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018200 con 0x7f63e0018390 2021-03-09T10:22:59.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:22:59.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:22:59.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00198e0 con 0x7f63e0018390 2021-03-09T10:22:59.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:22:59.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:22:59.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15621 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc03ff80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:22:59.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:22:59.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.867+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:22:59.881 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:22:59.871+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43262 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:22:59.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc049b00 con 0x7f63e0018390 2021-03-09T10:22:59.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.872+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:22:59.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.872+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:22:59.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.872+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:22:59.883 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:22:59.883 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:22:59.883 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:59.884 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:22:59.884 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:22:59.884 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:22:59.884 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:22:59.884 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:22:59.885 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:22:59.885 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:22:59.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc049b00 con 0x7f63e0018390 2021-03-09T10:22:59.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.872+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:22:59.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.872+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:22:59.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:22:59.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0497e0 con 0x7f63e0018390 2021-03-09T10:23:00.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:00.439+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:00.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:00.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:00.563+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:01.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:01.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:01.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:01.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:02.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:02.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:02.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:02.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:03.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:03.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:03.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:03.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:04.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:04.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:04.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:04.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:04.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15621 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc03ff80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:04.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15621 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc03ff80).stop 2021-03-09T10:23:04.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:04.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:04.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:04.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:04.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:23:04.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00190f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:04.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:04.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:04.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:04.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:04.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:04.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:04.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001230 con 0x7f63d8003400 2021-03-09T10:23:04.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:04.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065b90 con 0x7f63d8009130 2021-03-09T10:23:04.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:04.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00668f0 con 0x7f63d8003840 2021-03-09T10:23:04.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:04.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:04.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.872+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:04.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:04.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:04.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00190f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:04.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:23:04.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:04.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:04.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:04.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:04.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:04.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:04.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:04.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:23:04.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:23:04.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:04.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:04.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c110 con 0x7f63d8003840 2021-03-09T10:23:04.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:04.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:04.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63d8003840 2021-03-09T10:23:04.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:23:04.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.874+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:04.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.875+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:23:04.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.875+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:04.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.875+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00190f0 secure :-1 s=READY pgs=15634 cs=0 l=1 rev1=1 rx=0x7f63f007b650 tx=0x7f63f0041eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:04.886 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:04.878+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43302 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:04.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:23:04.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.878+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:04.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.878+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:04.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.878+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:04.888 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:04.888 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:04.889 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:04.889 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:04.889 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:04.889 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:04.890 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:04.890 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:04.890 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:04.890 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:04.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b490 con 0x7f63d8003840 2021-03-09T10:23:04.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.878+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:04.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.878+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:23:04.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:04.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038d60 con 0x7f63d8003840 2021-03-09T10:23:05.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:05.440+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:05.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:05.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:05.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:05.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:06.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:06.563+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:06.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:06.563+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:07.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:07.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:07.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:07.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:08.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:08.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:08.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:08.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:09.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:09.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:09.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00190f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:09.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00190f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:09.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00190f0 secure :-1 s=READY pgs=15634 cs=0 l=1 rev1=1 rx=0x7f63f007b650 tx=0x7f63f0041eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:09.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00190f0 secure :-1 s=READY pgs=15634 cs=0 l=1 rev1=1 rx=0x7f63f007b650 tx=0x7f63f0041eb0).stop 2021-03-09T10:23:09.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:09.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:09.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00190f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:09.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00190f0 unknown :-1 s=CLOSED pgs=15634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:09.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:23:09.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:09.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:09.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:09.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:09.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:09.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:09.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:09.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009130 con 0x7f63d8003400 2021-03-09T10:23:09.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:09.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:23:09.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:09.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63e0018390 2021-03-09T10:23:09.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:09.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:09.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:09.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:09.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:23:09.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:09.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:09.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:09.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:09.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:09.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:09.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:09.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:09.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:09.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:09.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:09.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:09.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:23:09.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:23:09.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:09.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:09.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0018390 2021-03-09T10:23:09.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:09.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:09.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c110 con 0x7f63e0018390 2021-03-09T10:23:09.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:23:09.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.880+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:09.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.882+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:23:09.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.882+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:09.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.882+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15653 cs=0 l=1 rev1=1 rx=0x7f63f007f6d0 tx=0x7f63f003e1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:09.894 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:09.884+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43350 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:09.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b490 con 0x7f63e0018390 2021-03-09T10:23:09.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.885+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:09.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.885+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:09.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.885+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:09.896 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:09.897 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:09.897 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:09.897 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:09.897 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:09.897 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:09.898 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:09.898 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63e0018390 2021-03-09T10:23:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.888+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.888+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:23:09.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:09.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0073340 con 0x7f63e0018390 2021-03-09T10:23:10.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:10.441+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:10.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:10.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:10.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:10.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:10.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:10.564+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:11.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:11.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:11.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:11.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:12.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:12.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:12.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:12.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:13.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:13.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:13.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:13.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:14.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:14.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:14.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:14.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:14.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15653 cs=0 l=1 rev1=1 rx=0x7f63f007f6d0 tx=0x7f63f003e1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:14.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15653 cs=0 l=1 rev1=1 rx=0x7f63f007f6d0 tx=0x7f63f003e1f0).stop 2021-03-09T10:23:14.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:14.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:14.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:14.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:14.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:23:14.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:14.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:14.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:14.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:14.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:14.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:14.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:14.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065b90 con 0x7f63d8003400 2021-03-09T10:23:14.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:14.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9d0 con 0x7f63d8009130 2021-03-09T10:23:14.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:14.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f040 con 0x7f63d8003840 2021-03-09T10:23:14.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:14.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:14.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:14.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:14.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:23:14.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:14.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:14.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:14.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.885+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:14.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.886+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:14.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.886+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:14.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.886+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:14.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:14.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:14.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:14.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:14.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:14.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:23:14.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:23:14.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:14.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:14.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00679c0 con 0x7f63d8003840 2021-03-09T10:23:14.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:14.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:14.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63d8003840 2021-03-09T10:23:14.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:23:14.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:14.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15664 cs=0 l=1 rev1=1 rx=0x7f63f00298d0 tx=0x7f63f0042df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:14.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:23:14.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.887+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:14.901 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:14.891+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43390 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:14.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b490 con 0x7f63d8003840 2021-03-09T10:23:14.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.892+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:14.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.892+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:14.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.892+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:14.903 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:14.903 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:14.903 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:14.903 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:14.904 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:14.904 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:14.904 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:14.905 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:14.905 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:14.905 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:14.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:23:14.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.895+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:14.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.895+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:23:14.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:14.896+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007cc20 con 0x7f63d8003840 2021-03-09T10:23:15.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:15.442+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:15.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:15.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:15.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:15.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:16.564+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:16.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:16.564+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:17.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:17.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:17.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:17.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:18.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:18.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:18.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:18.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:19.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:19.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:19.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15664 cs=0 l=1 rev1=1 rx=0x7f63f00298d0 tx=0x7f63f0042df0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15664 cs=0 l=1 rev1=1 rx=0x7f63f00298d0 tx=0x7f63f0042df0).stop 2021-03-09T10:23:19.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:19.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:19.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:19.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=CLOSED pgs=15664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:19.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:23:19.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:19.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:19.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:19.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:19.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:19.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:19.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:19.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00668f0 con 0x7f63d8003400 2021-03-09T10:23:19.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:19.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:23:19.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:19.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ddf0 con 0x7f63e0018390 2021-03-09T10:23:19.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:19.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:19.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:19.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:19.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:19.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:19.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:23:19.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:19.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.892+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:19.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:19.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:19.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:19.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:19.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:19.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:19.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:19.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:19.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:23:19.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:23:19.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:19.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:19.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007bf80 con 0x7f63e0018390 2021-03-09T10:23:19.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:19.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:19.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018200 con 0x7f63e0018390 2021-03-09T10:23:19.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:23:19.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.893+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:19.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.894+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:23:19.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.894+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:19.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.894+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15676 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc0039e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:19.908 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:19.897+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43428 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:19.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:23:19.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.898+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:19.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.898+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:19.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.898+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:19.910 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:19.910 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:19.910 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:19.910 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:19.911 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:19.911 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:19.911 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:19.911 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:19.912 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:19.912 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:19.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0711f0 con 0x7f63e0018390 2021-03-09T10:23:19.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.898+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:19.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.898+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:23:19.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:19.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc05eb80 con 0x7f63e0018390 2021-03-09T10:23:20.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:20.443+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:20.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:20.565+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:20.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:20.565+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:20.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:20.565+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:21.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:21.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:21.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:22.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:22.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:22.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:22.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:23.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:23.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:23.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:23.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:24.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:24.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:24.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:24.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:24.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15676 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc0039e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:24.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15676 cs=0 l=1 rev1=1 rx=0x7f63fc009130 tx=0x7f63fc0039e0).stop 2021-03-09T10:23:24.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:24.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:24.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:24.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:24.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:23:24.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:24.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:24.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:24.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:24.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:24.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:24.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:24.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:23:24.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:24.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000e4d0 con 0x7f63d8009130 2021-03-09T10:23:24.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:24.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:23:24.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:24.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:24.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:24.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:24.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.898+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:24.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.899+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:24.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.899+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:23:24.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.899+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:24.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.899+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:24.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.899+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:24.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.899+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:24.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.899+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:24.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.899+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:24.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:24.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:24.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:24.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:24.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:23:24.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:23:24.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:24.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:24.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d530 con 0x7f63d8003840 2021-03-09T10:23:24.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:24.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:24.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007bf80 con 0x7f63d8003840 2021-03-09T10:23:24.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:23:24.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.900+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:24.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.901+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:23:24.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.901+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 secure :-1 s=READY pgs=15692 cs=0 l=1 rev1=1 rx=0x7f63e005e840 tx=0x7f63fc008070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:24.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.901+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:24.914 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:24.905+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43468 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:24.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0499f0 con 0x7f63d8003840 2021-03-09T10:23:24.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.906+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:24.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.906+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:24.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.906+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:24.916 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:24.916 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:24.916 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:24.916 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:24.917 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:24.917 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:24.917 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:24.917 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:24.918 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:24.918 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:24.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.908+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051a10 con 0x7f63d8003840 2021-03-09T10:23:24.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.908+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:24.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.908+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:23:24.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:24.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0499f0 con 0x7f63d8003840 2021-03-09T10:23:25.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:25.444+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:25.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:25.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:25.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:25.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:26.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:26.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:26.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:26.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:27.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:27.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:27.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:27.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:28.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:28.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:28.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:28.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:29.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:29.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:29.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:29.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:29.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 secure :-1 s=READY pgs=15692 cs=0 l=1 rev1=1 rx=0x7f63e005e840 tx=0x7f63fc008070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:29.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 secure :-1 s=READY pgs=15692 cs=0 l=1 rev1=1 rx=0x7f63e005e840 tx=0x7f63fc008070).stop 2021-03-09T10:23:29.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:29.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:29.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:29.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 unknown :-1 s=CLOSED pgs=15692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:29.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:23:29.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:29.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:29.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:29.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:29.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:29.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:29.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:29.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012550 con 0x7f63d8003400 2021-03-09T10:23:29.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:29.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9d0 con 0x7f63d8009130 2021-03-09T10:23:29.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:29.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001cb0 con 0x7f63e0018390 2021-03-09T10:23:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:29.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:29.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:29.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.907+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:29.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.907+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:23:29.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.907+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:29.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.907+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:29.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.907+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:29.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.907+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:29.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.907+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:29.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:29.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:29.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:29.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:29.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:29.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:23:29.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:23:29.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:29.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:29.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024210 con 0x7f63e0018390 2021-03-09T10:23:29.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:29.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:29.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d530 con 0x7f63e0018390 2021-03-09T10:23:29.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:23:29.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.908+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:29.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.909+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15707 cs=0 l=1 rev1=1 rx=0x7f63e0022430 tx=0x7f63fc02d840).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:29.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.909+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:23:29.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.909+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:29.922 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:29.913+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43506 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:29.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052e80 con 0x7f63e0018390 2021-03-09T10:23:29.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.914+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:29.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.914+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:29.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.914+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:29.924 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:29.924 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:29.925 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:29.925 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:29.925 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:29.925 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:29.925 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:29.926 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:29.926 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:29.926 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:29.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.916+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052e80 con 0x7f63e0018390 2021-03-09T10:23:29.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.916+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:29.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.916+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:23:29.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:29.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0497e0 con 0x7f63e0018390 2021-03-09T10:23:30.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:30.445+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:30.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:30.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:30.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:30.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:30.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:30.567+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:31.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:31.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:31.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:31.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:32.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:32.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:32.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:32.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:33.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:33.567+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:33.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:33.567+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:34.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:34.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:34.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:34.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:34.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15707 cs=0 l=1 rev1=1 rx=0x7f63e0022430 tx=0x7f63fc02d840).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:34.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15707 cs=0 l=1 rev1=1 rx=0x7f63e0022430 tx=0x7f63fc02d840).stop 2021-03-09T10:23:34.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:34.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:34.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:34.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:34.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:23:34.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:34.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:34.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:34.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:34.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:34.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:34.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:34.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e500 con 0x7f63d8003400 2021-03-09T10:23:34.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:34.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:23:34.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:34.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d920 con 0x7f63d8003840 2021-03-09T10:23:34.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:34.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:34.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:34.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:34.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:34.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.915+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:34.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.915+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:23:34.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.915+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:34.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.915+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:34.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:34.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:34.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:34.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:34.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:34.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:34.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:34.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:34.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:23:34.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:23:34.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:34.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:34.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da60 con 0x7f63d8003840 2021-03-09T10:23:34.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:34.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:34.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024210 con 0x7f63d8003840 2021-03-09T10:23:34.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:23:34.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:34.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:23:34.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:34.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.917+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000d470 secure :-1 s=READY pgs=15725 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc078720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:34.941 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:34.921+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43546 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:34.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc078870 con 0x7f63d8003840 2021-03-09T10:23:34.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.921+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:34.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.921+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:34.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.921+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:34.944 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:34.945 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:34.945 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:34.946 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:34.947 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:34.947 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:34.948 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:34.948 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:34.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc078870 con 0x7f63d8003840 2021-03-09T10:23:34.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.924+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:34.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.924+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:23:34.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:34.925+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065410 con 0x7f63d8003840 2021-03-09T10:23:35.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:35.446+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:35.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:35.566+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:35.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:35.566+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:36.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:36.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:36.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:36.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:37.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:37.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:37.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:37.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:38.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:38.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:38.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:38.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:39.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:39.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:39.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:39.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:39.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000d470 secure :-1 s=READY pgs=15725 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc078720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:39.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000d470 secure :-1 s=READY pgs=15725 cs=0 l=1 rev1=1 rx=0x7f63fc076040 tx=0x7f63fc078720).stop 2021-03-09T10:23:39.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:39.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:39.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000d470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:39.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000d470 unknown :-1 s=CLOSED pgs=15725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:39.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:23:39.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:39.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:39.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:39.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:39.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:39.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:39.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:39.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:23:39.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:39.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065dd0 con 0x7f63d8009130 2021-03-09T10:23:39.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:39.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001cb0 con 0x7f63e0018390 2021-03-09T10:23:39.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:39.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:39.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:39.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:39.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:39.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:39.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:23:39.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:39.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.922+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:39.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:39.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:39.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:39.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:39.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:39.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:39.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:39.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:39.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:23:39.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:23:39.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018200 con 0x7f63e0018390 2021-03-09T10:23:39.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:39.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:39.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.923+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00679c0 con 0x7f63e0018390 2021-03-09T10:23:39.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.924+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:23:39.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.924+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:39.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.924+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:23:39.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.924+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:39.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.924+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15733 cs=0 l=1 rev1=1 rx=0x7f63f0057700 tx=0x7f63f0041e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:39.941 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:39.927+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43574 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T10:23:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.928+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.928+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:39.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.928+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:39.943 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:39.943 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:39.943 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:39.943 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:39.944 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:39.944 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:39.944 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:39.945 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:39.945 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:39.945 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:39.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.930+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T10:23:39.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.930+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:39.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.930+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:23:39.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:39.931+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0073da0 con 0x7f63e0018390 2021-03-09T10:23:40.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:40.447+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:40.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:40.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:40.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:40.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:40.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:40.568+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:41.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:41.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:41.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:41.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:42.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:42.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:42.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:42.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:43.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:43.568+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:43.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:43.568+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:44.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:44.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:44.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:44.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.928+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15733 cs=0 l=1 rev1=1 rx=0x7f63f0057700 tx=0x7f63f0041e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.928+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15733 cs=0 l=1 rev1=1 rx=0x7f63f0057700 tx=0x7f63f0041e90).stop 2021-03-09T10:23:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:44.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:44.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:44.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:23:44.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:44.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:44.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:44.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9d0 con 0x7f63d8003400 2021-03-09T10:23:44.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:44.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008fd0 con 0x7f63d8009130 2021-03-09T10:23:44.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:44.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001470 con 0x7f63d8003840 2021-03-09T10:23:44.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:44.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:44.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:44.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:44.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:44.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:44.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.931+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:23:44.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.931+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:44.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.931+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:44.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.932+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:44.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.932+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:44.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.932+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:44.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.932+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:44.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:44.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:44.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:44.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:44.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:23:44.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:23:44.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:44.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:44.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00212f0 con 0x7f63d8003840 2021-03-09T10:23:44.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:44.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:44.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:23:44.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:23:44.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:44.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020e00 secure :-1 s=READY pgs=15754 cs=0 l=1 rev1=1 rx=0x7f63f003e1c0 tx=0x7f63f0028770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:44.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:23:44.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.933+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:44.946 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:44.935+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43620 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:44.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63d8003840 2021-03-09T10:23:44.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.936+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:44.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.936+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:44.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.936+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:44.948 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:44.948 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:44.948 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:44.949 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:44.949 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:44.949 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:44.949 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:44.950 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:44.950 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:44.950 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:44.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0088c60 con 0x7f63d8003840 2021-03-09T10:23:44.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.936+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:44.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.936+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:23:44.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:44.938+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:23:45.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:45.448+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:45.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:45.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:45.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:45.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:46.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:46.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:46.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:46.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:47.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:47.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:47.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:47.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:48.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:48.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:48.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:48.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:49.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:49.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:49.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0020e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:49.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0020e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:49.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020e00 secure :-1 s=READY pgs=15754 cs=0 l=1 rev1=1 rx=0x7f63f003e1c0 tx=0x7f63f0028770).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:49.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020e00 secure :-1 s=READY pgs=15754 cs=0 l=1 rev1=1 rx=0x7f63f003e1c0 tx=0x7f63f0028770).stop 2021-03-09T10:23:49.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:49.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:49.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0020e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:49.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0020e00 unknown :-1 s=CLOSED pgs=15754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:49.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:23:49.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:49.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:49.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:49.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:49.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:49.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:49.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:49.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:23:49.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:49.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c550 con 0x7f63d8009130 2021-03-09T10:23:49.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:49.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0018390 2021-03-09T10:23:49.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.935+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:49.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:49.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:49.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:49.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:49.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.936+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:49.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.936+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:23:49.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.936+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:49.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.936+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:49.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.937+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:49.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.937+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:49.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.937+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:49.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:49.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:49.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:49.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:49.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:49.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:23:49.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:23:49.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:49.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:49.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018200 con 0x7f63e0018390 2021-03-09T10:23:49.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:49.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:49.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T10:23:49.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:23:49.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:49.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006d020 secure :-1 s=READY pgs=15761 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:49.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:23:49.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.938+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:49.949 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:49.941+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43658 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:49.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058000 con 0x7f63e0018390 2021-03-09T10:23:49.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.942+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:49.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.943+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:49.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.943+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:49.951 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:49.951 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:49.952 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:49.952 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:49.952 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:49.952 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:49.952 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:49.953 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:49.953 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:49.953 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:49.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058000 con 0x7f63e0018390 2021-03-09T10:23:49.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.945+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:49.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.945+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:23:49.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:49.947+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:23:50.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:50.449+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:50.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:50.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:50.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:50.569+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:50.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:50.569+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:51.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:51.569+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:51.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:51.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:52.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:52.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:52.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:52.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:53.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:53.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:53.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:53.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:54.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:54.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:54.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:23:54.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:54.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006d020 secure :-1 s=READY pgs=15761 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:54.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006d020 secure :-1 s=READY pgs=15761 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc04a740).stop 2021-03-09T10:23:54.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:54.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:54.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:54.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006d020 unknown :-1 s=CLOSED pgs=15761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:54.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:23:54.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00104f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:54.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:54.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:54.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008fd0 con 0x7f63d8003400 2021-03-09T10:23:54.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:54.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:23:54.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bc90 con 0x7f63d8003840 2021-03-09T10:23:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:54.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:54.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:54.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.942+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:54.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00104f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:54.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:23:54.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:54.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:54.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:54.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:54.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:23:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:23:54.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:54.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:54.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017920 con 0x7f63d8003840 2021-03-09T10:23:54.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:54.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:54.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:23:54.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:23:54.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.943+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:54.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.944+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00104f0 secure :-1 s=READY pgs=15773 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc03ff10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:54.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.944+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:23:54.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.944+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:54.956 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:54.949+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43697 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:54.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012e60 con 0x7f63d8003840 2021-03-09T10:23:54.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.950+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:54.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.950+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:54.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.950+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:54.957 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:54.958 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:54.958 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:54.958 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:54.958 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:54.959 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:54.959 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:54.959 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:54.959 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:54.960 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:54.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f63d8003840 2021-03-09T10:23:54.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.950+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:54.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.950+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:23:54.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:54.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8003840 2021-03-09T10:23:55.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:55.450+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:23:55.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:55.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:55.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:55.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:56.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:56.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:56.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:56.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:57.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:57.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:57.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:57.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:58.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:58.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:58.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:58.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:59.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.570+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:23:59.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.570+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:59.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:23:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:23:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00104f0 secure :-1 s=READY pgs=15773 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc03ff10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:23:59.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00104f0 secure :-1 s=READY pgs=15773 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc03ff10).stop 2021-03-09T10:23:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:23:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00104f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:23:59.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00104f0 unknown :-1 s=CLOSED pgs=15773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:23:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:23:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:59.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:23:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:23:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:23:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:23:59.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:59.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068490 con 0x7f63d8003400 2021-03-09T10:23:59.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:59.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c9e0 con 0x7f63d8009130 2021-03-09T10:23:59.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:23:59.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.949+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e0018390 2021-03-09T10:23:59.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.950+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:23:59.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.950+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:23:59.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.950+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.950+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:23:59.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.950+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:23:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.951+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:23:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.951+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:23:59.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.951+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:23:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.951+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:23:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.952+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:23:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.952+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:23:59.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.952+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:23:59.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:23:59.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:59.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:59.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:23:59.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:23:59.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:23:59.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:23:59.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:23:59.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:59.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018200 con 0x7f63e0018390 2021-03-09T10:23:59.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:23:59.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:23:59.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00212f0 con 0x7f63e0018390 2021-03-09T10:23:59.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:23:59.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.953+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:23:59.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.954+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15795 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:23:59.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.954+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:23:59.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.954+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:23:59.968 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:23:59.956+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43736 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:23:59.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:23:59.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.957+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:23:59.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.957+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:23:59.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.957+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:23:59.970 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:23:59.970 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:23:59.970 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:59.970 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:23:59.971 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:23:59.971 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:23:59.971 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:23:59.971 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:23:59.972 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:23:59.972 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:23:59.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:23:59.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.961+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:23:59.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.961+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:23:59.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:23:59.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042df0 con 0x7f63e0018390 2021-03-09T10:24:00.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:00.451+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:00.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:00.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:00.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:00.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:00.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:00.571+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:01.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:01.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:01.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:01.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:02.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:02.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:02.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:02.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:03.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:03.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:03.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:03.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:04.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:04.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:04.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:04.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:04.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15795 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:04.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15795 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004db20).stop 2021-03-09T10:24:04.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:04.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:04.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:04.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15795 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:04.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:24:04.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:04.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:04.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:04.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:04.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:04.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:04.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:04.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:24:04.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:04.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a070 con 0x7f63d8009130 2021-03-09T10:24:04.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:04.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:24:04.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:04.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:04.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:04.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:04.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.957+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:04.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.958+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:04.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.958+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:24:04.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.958+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:04.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.958+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:04.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.958+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:04.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.958+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:04.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.958+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:04.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:04.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:04.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:04.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:04.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:04.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:24:04.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:24:04.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da60 con 0x7f63d8003840 2021-03-09T10:24:04.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:04.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:04.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.959+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:24:04.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.960+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:24:04.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.960+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:04.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15807 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007a640).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:04.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.960+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:24:04.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.960+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:04.971 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:04.962+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43775 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:04.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:24:04.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.963+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:04.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.963+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:04.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.963+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:04.973 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:04.973 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:04.973 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:04.974 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:04.974 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:04.974 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:04.974 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:04.975 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:04.975 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:04.975 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:04.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.965+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:24:04.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.965+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:04.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.965+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:24:04.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:04.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f63d8003840 2021-03-09T10:24:05.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:05.452+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:05.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:05.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:05.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:05.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:06.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:06.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:06.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:06.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:07.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:07.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:07.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:07.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:08.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:08.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:08.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:08.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:09.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:09.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.962+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:09.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.962+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:09.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.962+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15807 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007a640).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:09.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.962+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15807 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007a640).stop 2021-03-09T10:24:09.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:09.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:09.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:09.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=15807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:09.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:24:09.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:09.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012550 con 0x7f63d8003400 2021-03-09T10:24:09.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064db0 con 0x7f63d8009130 2021-03-09T10:24:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:09.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e0018390 2021-03-09T10:24:09.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:09.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:09.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:09.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:24:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.963+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:09.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:09.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:09.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:09.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:09.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:24:09.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:24:09.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:09.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:09.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63e0018390 2021-03-09T10:24:09.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:09.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:09.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T10:24:09.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:24:09.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.964+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:09.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.965+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:24:09.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.965+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:09.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.965+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15819 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0031f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:09.981 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:09.968+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43814 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:24:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.969+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:09.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.969+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.969+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:09.983 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:09.984 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:09.985 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:09.985 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:24:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.973+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:09.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.973+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:24:09.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:09.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007a360 con 0x7f63e0018390 2021-03-09T10:24:10.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:10.452+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:10.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:10.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:10.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:10.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:10.571+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:11.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:11.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:11.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:11.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:12.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:12.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:12.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:13.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:13.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:13.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:13.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:14.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.571+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:14.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.571+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:14.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:15.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:15.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15819 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0031f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:15.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15819 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0031f50).stop 2021-03-09T10:24:15.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:15.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:15.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:15.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:15.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:24:15.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:15.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:15.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:15.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:15.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:15.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:15.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:15.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008fd0 con 0x7f63d8003400 2021-03-09T10:24:15.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:15.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:24:15.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:15.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fe20 con 0x7f63d8003840 2021-03-09T10:24:15.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:15.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:15.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:15.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:15.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.969+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:15.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:15.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.970+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:24:15.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.970+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:15.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.970+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:15.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.971+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:15.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.971+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:15.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.971+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:15.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:15.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:15.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:15.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:15.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:15.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:24:15.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:24:15.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:15.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:15.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024210 con 0x7f63d8003840 2021-03-09T10:24:15.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:15.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:15.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:24:15.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:24:15.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.972+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:15.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.973+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:24:15.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.973+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:15.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.973+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15837 cs=0 l=1 rev1=1 rx=0x7f63f00555f0 tx=0x7f63f0055730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:15.058 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:14.976+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43853 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:15.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.977+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:24:15.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.977+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:15.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.977+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:15.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.977+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:15.061 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:15.062 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:15.062 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:15.063 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:15.063 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:15.063 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:15.064 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:15.064 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:15.065 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:15.065 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:15.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:24:15.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.982+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:15.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.982+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:24:15.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:14.983+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038b10 con 0x7f63d8003840 2021-03-09T10:24:15.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:15.453+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:15.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:15.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:15.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:15.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:16.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:16.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:16.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:16.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:17.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:17.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:17.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:17.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:18.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:18.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:18.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:18.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:19.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:19.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:19.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:24:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15837 cs=0 l=1 rev1=1 rx=0x7f63f00555f0 tx=0x7f63f0055730).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:19.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=15837 cs=0 l=1 rev1=1 rx=0x7f63f00555f0 tx=0x7f63f0055730).stop 2021-03-09T10:24:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=15837 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:19.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:24:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:19.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:19.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:19.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:19.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:24:19.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:19.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:24:19.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:19.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0018390 2021-03-09T10:24:19.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:19.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:19.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:19.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:19.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:19.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:19.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:24:19.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:19.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.978+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:19.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:19.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:19.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:19.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:19.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:19.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:19.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:19.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:19.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:24:19.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:24:19.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:19.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:19.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00656e0 con 0x7f63e0018390 2021-03-09T10:24:19.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:19.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:19.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024210 con 0x7f63e0018390 2021-03-09T10:24:19.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:24:19.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.979+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:19.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.980+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15842 cs=0 l=1 rev1=1 rx=0x7f63f00578a0 tx=0x7f63f0055660).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:19.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.980+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:24:19.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.980+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:19.993 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:19.983+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43890 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:19.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63e0018390 2021-03-09T10:24:19.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.984+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:19.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.984+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:19.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.984+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:19.995 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:19.995 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:19.995 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:19.995 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:19.996 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:19.996 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:19.996 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:19.996 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:19.996 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:19.997 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:19.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.985+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:24:19.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.985+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:19.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.985+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:24:19.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:19.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63e0018390 2021-03-09T10:24:20.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:20.454+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:20.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:20.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:20.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:20.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:20.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:20.572+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:21.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:21.572+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:21.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:21.572+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:22.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:22.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:22.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:22.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:23.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:23.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:23.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:23.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:24.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:24.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:24.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:24.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:24.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15842 cs=0 l=1 rev1=1 rx=0x7f63f00578a0 tx=0x7f63f0055660).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:24.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15842 cs=0 l=1 rev1=1 rx=0x7f63f00578a0 tx=0x7f63f0055660).stop 2021-03-09T10:24:24.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:24.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:24.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:24.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:24.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:24:24.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:24.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:24.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:24.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:24.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:24.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:24.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:24.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008fd0 con 0x7f63d8003400 2021-03-09T10:24:24.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:24.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:24:24.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:24.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c160 con 0x7f63d8003840 2021-03-09T10:24:24.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:24.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:24.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:24.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:24.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.984+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:24.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.985+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:24.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.985+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:24:24.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.985+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:24.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.985+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:24.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.985+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:24.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.985+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:24.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.985+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:24.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:24.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:24.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:24.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:24.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:24.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:24:24.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:24:24.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:24.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:24.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da60 con 0x7f63d8003840 2021-03-09T10:24:24.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:24.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:24.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017920 con 0x7f63d8003840 2021-03-09T10:24:24.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:24:24.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.986+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:24.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.987+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15857 cs=0 l=1 rev1=1 rx=0x7f63e001b310 tx=0x7f63fc075fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:24.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.987+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:24:24.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.987+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:24.999 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:24.990+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43931 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:24.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63d8003840 2021-03-09T10:24:25.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.990+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:25.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.990+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:25.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.990+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:25.000 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:25.001 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:25.001 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:25.001 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:25.001 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:25.002 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:25.002 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:25.002 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:25.002 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:25.002 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:25.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63d8003840 2021-03-09T10:24:25.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.993+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:25.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.993+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:24:25.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:24.994+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:24:25.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:25.455+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:25.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:25.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:25.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:25.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:26.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:26.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:26.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:26.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:27.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:27.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:27.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:27.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:28.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:28.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:28.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:28.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:29.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.573+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:29.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.573+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:29.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:24:29.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:29.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15857 cs=0 l=1 rev1=1 rx=0x7f63e001b310 tx=0x7f63fc075fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:29.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=15857 cs=0 l=1 rev1=1 rx=0x7f63e001b310 tx=0x7f63fc075fe0).stop 2021-03-09T10:24:29.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:29.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:29.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:29.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=CLOSED pgs=15857 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:29.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:24:29.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:29.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:29.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:29.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:29.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:29.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:29.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:29.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9d0 con 0x7f63d8003400 2021-03-09T10:24:29.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:29.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:24:29.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:29.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c100 con 0x7f63e0018390 2021-03-09T10:24:29.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:29.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:29.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.990+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:29.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.991+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:29.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.991+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:29.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.991+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:29.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.991+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:24:29.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.991+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:30.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.991+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:30.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.991+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:30.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.991+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:30.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.991+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:30.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:30.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:30.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:30.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:30.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:30.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:24:30.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:24:30.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:30.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:30.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d8d0 con 0x7f63e0018390 2021-03-09T10:24:30.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:30.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:30.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00656e0 con 0x7f63e0018390 2021-03-09T10:24:30.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:24:30.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:30.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:24:30.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15869 cs=0 l=1 rev1=1 rx=0x7f63f0039a30 tx=0x7f63f0031e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:30.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.992+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:30.006 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:29.997+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 43970 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:30.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63e0018390 2021-03-09T10:24:30.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.998+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:30.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.998+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:30.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.998+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:30.007 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:30.008 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:30.008 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:30.008 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:30.008 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:30.008 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:30.009 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:30.009 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:30.009 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:30.009 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:30.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63e0018390 2021-03-09T10:24:30.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.998+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:30.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:29.998+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:24:30.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:30.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:24:30.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:30.456+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:30.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:30.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:30.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:30.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:30.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:30.574+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:31.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:31.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:31.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:31.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:32.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:32.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:32.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:32.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:33.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:33.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:33.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:33.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:34.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:34.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:35.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:35.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:35.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15869 cs=0 l=1 rev1=1 rx=0x7f63f0039a30 tx=0x7f63f0031e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:35.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15869 cs=0 l=1 rev1=1 rx=0x7f63f0039a30 tx=0x7f63f0031e60).stop 2021-03-09T10:24:35.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:35.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:35.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:35.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:35.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:24:35.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:35.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:35.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:35.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:35.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:35.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:35.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:35.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:24:35.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:35.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:24:35.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:35.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022150 con 0x7f63d8003840 2021-03-09T10:24:35.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:35.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:35.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:35.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:35.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:35.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:35.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:24:35.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:35.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.998+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:35.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.999+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:35.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.999+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:35.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:34.999+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:35.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:35.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:35.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:35.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:35.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:35.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:24:35.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:24:35.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:35.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:35.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:24:35.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:35.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d8d0 con 0x7f63d8003840 2021-03-09T10:24:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:24:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.000+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.001+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=15884 cs=0 l=1 rev1=1 rx=0x7f63f00066c0 tx=0x7f63f00235e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:35.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.001+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:24:35.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.001+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:35.013 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:35.005+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44009 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:35.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00296c0 con 0x7f63d8003840 2021-03-09T10:24:35.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.006+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:35.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.006+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:35.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.006+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:35.015 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:35.015 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:35.015 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:35.015 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:35.016 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:35.016 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:35.016 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:35.016 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:35.016 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:35.017 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:35.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:24:35.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.008+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:35.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.008+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:24:35.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:24:35.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.457+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:35.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:35.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:35.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:36.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:36.574+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:36.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:36.574+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:37.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:37.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:37.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:37.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:38.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:38.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:38.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:38.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:39.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:39.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:39.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:39.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:40.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.005+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:40.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.005+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:40.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.005+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=15884 cs=0 l=1 rev1=1 rx=0x7f63f00066c0 tx=0x7f63f00235e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:40.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.005+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=15884 cs=0 l=1 rev1=1 rx=0x7f63f00066c0 tx=0x7f63f00235e0).stop 2021-03-09T10:24:40.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:40.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:40.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:40.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=CLOSED pgs=15884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:40.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:24:40.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:40.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:40.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:40.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:40.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:40.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:40.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:40.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008fd0 con 0x7f63d8003400 2021-03-09T10:24:40.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:40.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:24:40.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:40.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e0018390 2021-03-09T10:24:40.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:40.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:40.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:40.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:40.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:40.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:40.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:24:40.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:40.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.006+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:40.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.007+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:40.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.007+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:40.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.007+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:40.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:40.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:40.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:40.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:40.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:40.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:24:40.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:24:40.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:40.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:40.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00656e0 con 0x7f63e0018390 2021-03-09T10:24:40.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:40.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:40.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63e0018390 2021-03-09T10:24:40.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:24:40.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:40.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15898 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f00066f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:40.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:24:40.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.008+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:40.026 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:40.011+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44048 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:40.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63e0018390 2021-03-09T10:24:40.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.012+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:40.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.012+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:40.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.012+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:40.028 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:40.028 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:40.028 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:40.028 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:40.029 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:40.029 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:40.029 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:40.029 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:40.030 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:40.030 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:40.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:24:40.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.016+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:40.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.016+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:24:40.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63e0018390 2021-03-09T10:24:40.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.458+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:40.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:40.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:40.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:40.575+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:41.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:41.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:41.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:41.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:42.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:42.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:42.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:42.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:43.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:43.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:43.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:43.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:44.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:44.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:44.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:44.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:45.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15898 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f00066f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15898 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f00066f0).stop 2021-03-09T10:24:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:45.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:45.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:45.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15898 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:45.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:24:45.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:45.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:45.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:45.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:45.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:45.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:45.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:45.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fe20 con 0x7f63d8003400 2021-03-09T10:24:45.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:45.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:24:45.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:45.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001afd0 con 0x7f63d8003840 2021-03-09T10:24:45.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:45.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:45.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:45.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:45.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:45.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:45.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.013+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:24:45.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.013+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:45.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.013+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:45.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.014+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:45.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.014+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:45.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.014+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:45.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:45.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:45.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:45.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:45.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:45.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:24:45.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:24:45.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:45.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:45.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:24:45.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:45.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:45.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00656e0 con 0x7f63d8003840 2021-03-09T10:24:45.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:24:45.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.015+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:45.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.016+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:24:45.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.016+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:45.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.016+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008e20 secure :-1 s=READY pgs=15917 cs=0 l=1 rev1=1 rx=0x7f63f0031e30 tx=0x7f63f00286b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:45.026 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:45.018+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44087 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:45.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028480 con 0x7f63d8003840 2021-03-09T10:24:45.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.019+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:45.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.019+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:45.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.019+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:45.028 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:45.029 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:45.029 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:45.029 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:45.029 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:45.029 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:45.030 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:45.030 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:45.030 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:45.030 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:45.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63d8003840 2021-03-09T10:24:45.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.024+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:45.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0057390 con 0x7f63d8003840 2021-03-09T10:24:45.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.024+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:24:45.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.459+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.575+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:45.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:45.575+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:46.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:46.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:46.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:46.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:47.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:47.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:47.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:47.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:48.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:48.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:48.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:48.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:49.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:49.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:49.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:49.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:50.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.018+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:50.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.018+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:50.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.018+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008e20 secure :-1 s=READY pgs=15917 cs=0 l=1 rev1=1 rx=0x7f63f0031e30 tx=0x7f63f00286b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:50.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.018+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008e20 secure :-1 s=READY pgs=15917 cs=0 l=1 rev1=1 rx=0x7f63f0031e30 tx=0x7f63f00286b0).stop 2021-03-09T10:24:50.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:50.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:50.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008e20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:50.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008e20 unknown :-1 s=CLOSED pgs=15917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:50.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:24:50.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:50.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:50.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:50.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:50.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:50.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:50.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:50.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:24:50.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:50.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac10 con 0x7f63d8009130 2021-03-09T10:24:50.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:50.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011d70 con 0x7f63e0018390 2021-03-09T10:24:50.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:50.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:50.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:50.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:50.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.019+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:50.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:50.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:24:50.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:50.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:50.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:50.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:50.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:50.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:50.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:50.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:50.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:50.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:50.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:24:50.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:24:50.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:50.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:50.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022100 con 0x7f63e0018390 2021-03-09T10:24:50.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:50.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:50.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T10:24:50.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:24:50.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.020+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:50.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.021+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15923 cs=0 l=1 rev1=1 rx=0x7f63e00697b0 tx=0x7f63fc0039e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:50.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.021+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:24:50.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.021+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:50.034 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:50.024+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44126 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:50.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0113b0 con 0x7f63e0018390 2021-03-09T10:24:50.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.024+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:50.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.024+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:50.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.024+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:50.036 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:50.036 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:50.036 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:50.037 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:50.037 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:50.037 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:50.037 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:50.038 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:50.038 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:50.038 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:50.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0018390 2021-03-09T10:24:50.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.025+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:50.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.025+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:24:50.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.026+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f4b0 con 0x7f63e0018390 2021-03-09T10:24:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.460+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:50.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:50.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:50.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:50.576+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:51.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:51.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:51.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:51.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:52.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:52.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:52.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:52.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:53.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:53.576+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:53.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:53.576+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:54.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:54.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:54.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:55.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:24:55.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:24:55.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15923 cs=0 l=1 rev1=1 rx=0x7f63e00697b0 tx=0x7f63fc0039e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:24:55.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15923 cs=0 l=1 rev1=1 rx=0x7f63e00697b0 tx=0x7f63fc0039e0).stop 2021-03-09T10:24:55.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:55.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:24:55.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:24:55.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15923 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:24:55.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:24:55.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:55.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:24:55.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:55.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:24:55.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:24:55.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:24:55.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:55.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:24:55.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:55.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:24:55.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:24:55.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c100 con 0x7f63d8003840 2021-03-09T10:24:55.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:24:55.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:24:55.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:55.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:24:55.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:24:55.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.026+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:24:55.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.026+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:24:55.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.026+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:24:55.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.026+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:24:55.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:24:55.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:24:55.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:24:55.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:24:55.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:55.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:55.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:24:55.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:24:55.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:24:55.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:24:55.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:24:55.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:55.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024210 con 0x7f63d8003840 2021-03-09T10:24:55.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:24:55.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:24:55.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:24:55.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:24:55.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.027+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:55.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.028+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:24:55.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.029+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:24:55.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.028+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=15944 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:24:55.041 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:24:55.031+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44163 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:24:55.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63d8003840 2021-03-09T10:24:55.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.033+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:24:55.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.033+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:24:55.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.033+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:24:55.043 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:24:55.043 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:24:55.043 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:55.043 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:24:55.044 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:24:55.044 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:24:55.044 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:24:55.044 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:24:55.044 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:24:55.045 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:24:55.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:24:55.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.033+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:24:55.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.033+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:24:55.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.037+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:24:55.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.461+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:24:55.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:55.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:55.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:56.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:56.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:56.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:57.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:57.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:57.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:57.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:58.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:58.577+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:58.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:58.577+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:24:59.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:59.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:24:59.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:24:59.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:00.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:00.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:00.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=15944 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:00.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=15944 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0000c30).stop 2021-03-09T10:25:00.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:00.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:00.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:00.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=CLOSED pgs=15944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:00.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:25:00.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:00.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:00.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:00.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:00.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:00.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:00.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:00.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:25:00.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:00.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:25:00.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:00.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00290c0 con 0x7f63e0018390 2021-03-09T10:25:00.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:00.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:00.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.031+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:00.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.032+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:00.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.032+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:00.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.033+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:00.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.033+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:25:00.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.033+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:00.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.033+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:00.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.034+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:00.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.034+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:00.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.034+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:00.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:00.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:00.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:00.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:00.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:00.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:25:00.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:25:00.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:00.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:00.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T10:25:00.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:00.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:00.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022100 con 0x7f63e0018390 2021-03-09T10:25:00.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:25:00.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:00.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:25:00.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:00.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.035+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15961 cs=0 l=1 rev1=1 rx=0x7f63fc0789b0 tx=0x7f63fc078820).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:00.051 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:00.037+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44204 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:00.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.038+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63e0018390 2021-03-09T10:25:00.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.038+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:00.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.038+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:00.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.038+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:00.053 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:00.053 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:00.053 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:00.053 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:00.054 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:00.054 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:00.054 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:00.054 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:00.054 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:00.055 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:00.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63e0018390 2021-03-09T10:25:00.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.041+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:00.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.041+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:25:00.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.043+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:25:00.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.462+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:00.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:00.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:00.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:00.578+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:01.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:01.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:01.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:01.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:02.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:02.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:02.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:02.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:03.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:03.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:03.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:03.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:04.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:04.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:04.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:04.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:05.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:05.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.038+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:05.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.038+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15961 cs=0 l=1 rev1=1 rx=0x7f63fc0789b0 tx=0x7f63fc078820).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:05.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.038+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15961 cs=0 l=1 rev1=1 rx=0x7f63fc0789b0 tx=0x7f63fc078820).stop 2021-03-09T10:25:05.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:05.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:05.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:05.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15961 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:05.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:25:05.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:05.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:05.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:05.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:05.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:05.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:05.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:05.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:25:05.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:05.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:25:05.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:05.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bc90 con 0x7f63d8003840 2021-03-09T10:25:05.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:05.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:05.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:05.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:05.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.039+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:05.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.040+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:05.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.040+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:25:05.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.040+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:05.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.040+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:05.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:05.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:05.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:05.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:05.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:05.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:05.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:05.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:05.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:25:05.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:25:05.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:05.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:05.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f050 con 0x7f63d8003840 2021-03-09T10:25:05.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:05.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:05.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da60 con 0x7f63d8003840 2021-03-09T10:25:05.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:25:05.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.041+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:05.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.042+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:25:05.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.042+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:05.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=15974 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc02b460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:05.055 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:05.044+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44243 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:05.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.044+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63d8003840 2021-03-09T10:25:05.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.044+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:05.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.044+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:05.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.044+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:05.057 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:05.057 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:05.057 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:05.057 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:05.058 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:05.058 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:05.058 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:05.058 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:05.058 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:05.059 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:05.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02b570 con 0x7f63d8003840 2021-03-09T10:25:05.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.048+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:05.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63d8003840 2021-03-09T10:25:05.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.049+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:25:05.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.463+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:05.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:05.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:06.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:06.578+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:06.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:06.578+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:07.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:07.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:07.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:07.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:08.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:08.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:08.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:09.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:09.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:09.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:09.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:10.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:25:10.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:10.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=15974 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc02b460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:10.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=15974 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc02b460).stop 2021-03-09T10:25:10.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:10.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:10.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:10.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=CLOSED pgs=15974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:10.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:25:10.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:10.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:10.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:10.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:10.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:10.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:10.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:10.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00286e0 con 0x7f63d8003400 2021-03-09T10:25:10.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:10.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001b50 con 0x7f63d8009130 2021-03-09T10:25:10.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:10.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c9e0 con 0x7f63e0018390 2021-03-09T10:25:10.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:10.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:10.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:10.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:10.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:10.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:10.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:25:10.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:10.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.045+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:10.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.046+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:10.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.046+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:10.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.046+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:10.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:10.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:10.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:10.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:10.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:10.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:25:10.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:25:10.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:10.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:10.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0014fc0 con 0x7f63e0018390 2021-03-09T10:25:10.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:10.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:10.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f050 con 0x7f63e0018390 2021-03-09T10:25:10.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:25:10.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:10.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15985 cs=0 l=1 rev1=1 rx=0x7f63e0001420 tx=0x7f63fc06de60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:10.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:25:10.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.047+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:10.060 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:10.051+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44282 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:10.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63e0018390 2021-03-09T10:25:10.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.051+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:10.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.051+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:10.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.051+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:10.062 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:10.062 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:10.062 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:10.062 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:10.063 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:10.063 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:10.063 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:10.063 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:10.063 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:10.064 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:10.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63e0018390 2021-03-09T10:25:10.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.055+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:10.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.055+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e0018390 2021-03-09T10:25:10.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.055+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:25:10.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.464+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:10.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:10.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:10.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:10.579+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:11.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:11.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:11.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:11.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:12.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:12.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:12.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:12.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:13.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:13.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:13.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:13.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:14.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:14.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:14.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:14.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:15.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:25:15.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:15.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15985 cs=0 l=1 rev1=1 rx=0x7f63e0001420 tx=0x7f63fc06de60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:15.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=15985 cs=0 l=1 rev1=1 rx=0x7f63e0001420 tx=0x7f63fc06de60).stop 2021-03-09T10:25:15.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:15.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:15.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:15.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=15985 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:15.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:25:15.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:15.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:15.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:15.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:15.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:15.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.051+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:15.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:15.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:25:15.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:15.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012550 con 0x7f63d8009130 2021-03-09T10:25:15.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:15.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019870 con 0x7f63d8003840 2021-03-09T10:25:15.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:15.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:15.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:15.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:15.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:15.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.053+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:15.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.053+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:25:15.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.053+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:15.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.053+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:15.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.053+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:15.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.053+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:15.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.053+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:15.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:15.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:15.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:15.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:15.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:15.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:25:15.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:25:15.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:15.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:15.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:25:15.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:15.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:15.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0014fc0 con 0x7f63d8003840 2021-03-09T10:25:15.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:25:15.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:15.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:25:15.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:15.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.055+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=16002 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc03de80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:15.067 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:15.058+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44321 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:15.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63d8003840 2021-03-09T10:25:15.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.059+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:15.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.059+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:15.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.059+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:15.069 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:15.069 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:15.070 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:15.070 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:15.070 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:15.070 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:15.070 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:15.071 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:15.071 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:15.071 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:15.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63d8003840 2021-03-09T10:25:15.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.065+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:15.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.065+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:25:15.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.066+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03d170 con 0x7f63d8003840 2021-03-09T10:25:15.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.465+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:15.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:15.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:15.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:16.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:16.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:16.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:16.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:17.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:17.579+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:17.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:17.579+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:18.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:18.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:18.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:18.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:19.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:19.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:19.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:19.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:20.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=16002 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc03de80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 secure :-1 s=READY pgs=16002 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc03de80).stop 2021-03-09T10:25:20.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bcb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bcb0 unknown :-1 s=CLOSED pgs=16002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:25:20.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:20.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:20.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:20.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:20.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:25:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:20.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011d70 con 0x7f63d8009130 2021-03-09T10:25:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018cb0 con 0x7f63e0018390 2021-03-09T10:25:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:20.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:20.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:20.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.060+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:20.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.060+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:20.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.060+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:20.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.060+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:25:20.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.060+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:20.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.060+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:20.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.060+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:20.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.060+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:20.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.060+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:20.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:20.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:20.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:20.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:20.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:20.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:25:20.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:25:20.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:20.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:20.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001da60 con 0x7f63e0018390 2021-03-09T10:25:20.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:20.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:20.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024210 con 0x7f63e0018390 2021-03-09T10:25:20.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:25:20.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.061+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:20.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.062+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16005 cs=0 l=1 rev1=1 rx=0x7f63f003e9f0 tx=0x7f63f004c2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:20.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.062+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:25:20.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.062+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:20.075 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:20.066+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44360 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:20.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:25:20.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.067+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:20.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.067+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:20.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.067+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:20.076 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:20.077 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:20.077 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:20.077 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:20.077 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:20.077 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:20.078 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:20.078 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:20.078 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:20.078 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:20.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:25:20.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.068+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:20.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.068+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:25:20.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63e0018390 2021-03-09T10:25:20.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.466+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:20.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:20.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:20.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:20.580+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:21.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:21.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:21.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:22.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:22.580+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:22.580+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:23.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:23.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:23.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:23.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:24.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:24.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:24.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:24.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:25.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:25.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:25.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16005 cs=0 l=1 rev1=1 rx=0x7f63f003e9f0 tx=0x7f63f004c2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:25.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16005 cs=0 l=1 rev1=1 rx=0x7f63f003e9f0 tx=0x7f63f004c2d0).stop 2021-03-09T10:25:25.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:25.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:25.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:25.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:25.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:25:25.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:25.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:25.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:25.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:25.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:25.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:25.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:25.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021ce0 con 0x7f63d8003400 2021-03-09T10:25:25.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:25.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9d0 con 0x7f63d8009130 2021-03-09T10:25:25.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:25.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019870 con 0x7f63d8003840 2021-03-09T10:25:25.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:25.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:25.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:25.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:25.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:25.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:25:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:25.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.067+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:25.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:25.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:25.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:25.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:25:25.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:25:25.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00303f0 con 0x7f63d8003840 2021-03-09T10:25:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:25.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001da60 con 0x7f63d8003840 2021-03-09T10:25:25.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:25:25.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.068+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:25.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.069+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:25:25.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.069+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:25.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.069+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c330 secure :-1 s=READY pgs=16019 cs=0 l=1 rev1=1 rx=0x7f63f0040000 tx=0x7f63f0041f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:25.081 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:25.073+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44399 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.074+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003ec30 con 0x7f63d8003840 2021-03-09T10:25:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.074+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.074+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:25.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.074+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:25.083 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:25.084 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:25.084 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:25.084 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:25.084 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:25.085 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:25.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.074+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007bc80 con 0x7f63d8003840 2021-03-09T10:25:25.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.074+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:25.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.074+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:25:25.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.075+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003ea20 con 0x7f63d8003840 2021-03-09T10:25:25.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.467+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:25.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:25.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:25.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:26.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:26.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:26.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:26.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:27.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:27.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:27.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:27.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:28.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:28.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:28.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:28.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:29.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:29.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:29.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:29.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:30.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:30.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:30.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c330 secure :-1 s=READY pgs=16019 cs=0 l=1 rev1=1 rx=0x7f63f0040000 tx=0x7f63f0041f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:30.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c330 secure :-1 s=READY pgs=16019 cs=0 l=1 rev1=1 rx=0x7f63f0040000 tx=0x7f63f0041f00).stop 2021-03-09T10:25:30.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:30.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:30.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002c330 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:30.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c330 unknown :-1 s=CLOSED pgs=16019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:30.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:25:30.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:30.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:30.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:30.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:30.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:30.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:30.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:30.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012550 con 0x7f63d8003400 2021-03-09T10:25:30.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:30.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008a90 con 0x7f63d8009130 2021-03-09T10:25:30.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:30.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:25:30.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:30.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:30.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:30.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:30.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:30.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.075+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:30.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.075+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:25:30.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.075+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:30.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.075+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:30.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.075+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:30.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.075+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:30.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.075+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:30.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:30.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:30.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:30.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:30.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:30.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:25:30.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:25:30.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:30.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:30.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00679f0 con 0x7f63e0018390 2021-03-09T10:25:30.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:30.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:30.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00303f0 con 0x7f63e0018390 2021-03-09T10:25:30.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:25:30.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:30.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16036 cs=0 l=1 rev1=1 rx=0x7f63e0001d20 tx=0x7f63f007be60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:30.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:25:30.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.076+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:30.098 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:30.080+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44438 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:30.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003ec30 con 0x7f63e0018390 2021-03-09T10:25:30.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.081+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:30.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.081+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:30.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.081+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:30.100 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:30.100 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:30.101 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:30.101 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:30.101 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:30.101 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:30.102 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:30.102 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:30.102 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:30.102 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:30.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.083+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00243f0 con 0x7f63e0018390 2021-03-09T10:25:30.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.083+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:30.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.083+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:25:30.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.084+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003ea20 con 0x7f63e0018390 2021-03-09T10:25:30.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.468+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:30.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:30.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:30.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:30.582+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:31.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:31.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:31.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:31.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:32.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:32.581+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:32.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:32.581+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:33.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:33.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:33.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:33.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:34.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:34.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:34.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:35.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:25:35.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:35.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16036 cs=0 l=1 rev1=1 rx=0x7f63e0001d20 tx=0x7f63f007be60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:35.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16036 cs=0 l=1 rev1=1 rx=0x7f63e0001d20 tx=0x7f63f007be60).stop 2021-03-09T10:25:35.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:35.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:35.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:35.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:35.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:25:35.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:35.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:35.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:35.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:35.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:35.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:35.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:35.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025c80 con 0x7f63d8003400 2021-03-09T10:25:35.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:35.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011d70 con 0x7f63d8009130 2021-03-09T10:25:35.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:35.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015b40 con 0x7f63d8003840 2021-03-09T10:25:35.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:35.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:35.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:35.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:35.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:35.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:35.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:25:35.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:35.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:35.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:35.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:35.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:35.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:35.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:35.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.082+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:35.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:35.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:35.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:25:35.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:25:35.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:35.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:35.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0009680 con 0x7f63d8003840 2021-03-09T10:25:35.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:35.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:35.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00679f0 con 0x7f63d8003840 2021-03-09T10:25:35.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:25:35.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.083+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:35.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.084+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:25:35.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.084+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:35.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.084+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029230 secure :-1 s=READY pgs=16046 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0066380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:35.099 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:35.087+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44475 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:35.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038470 con 0x7f63d8003840 2021-03-09T10:25:35.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.087+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:35.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.087+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:35.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.087+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:35.100 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:35.101 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:35.101 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:35.101 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:35.101 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:35.101 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:35.102 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:35.102 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:35.103 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:35.103 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:35.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:25:35.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.087+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:35.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.087+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:25:35.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.090+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63d8003840 2021-03-09T10:25:35.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.469+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:35.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.582+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:35.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:35.582+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:36.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:36.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:36.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:36.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:37.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:37.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:37.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:37.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:38.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:38.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:38.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:38.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:39.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:39.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:39.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:39.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:40.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:25:40.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:40.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029230 secure :-1 s=READY pgs=16046 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0066380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:40.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029230 secure :-1 s=READY pgs=16046 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0066380).stop 2021-03-09T10:25:40.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:40.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:40.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:40.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029230 unknown :-1 s=CLOSED pgs=16046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:40.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:25:40.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:40.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:40.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:40.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:40.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:40.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:40.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:40.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012550 con 0x7f63d8003400 2021-03-09T10:25:40.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:40.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007ae10 con 0x7f63d8009130 2021-03-09T10:25:40.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:40.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c9e0 con 0x7f63e0018390 2021-03-09T10:25:40.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:40.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:40.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:40.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:40.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:25:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:40.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.088+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.089+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.089+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.089+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:40.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:40.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:40.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:40.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:40.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:25:40.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:25:40.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:40.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:40.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024210 con 0x7f63e0018390 2021-03-09T10:25:40.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:40.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0009680 con 0x7f63e0018390 2021-03-09T10:25:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:25:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.090+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.091+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16065 cs=0 l=1 rev1=1 rx=0x7f63f00738e0 tx=0x7f63f0039e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:40.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.091+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:25:40.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.091+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:40.105 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:40.094+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44516 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0073520 con 0x7f63e0018390 2021-03-09T10:25:40.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.095+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:40.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.095+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:40.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.095+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:40.107 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:40.107 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:40.108 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:40.108 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:40.108 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:40.108 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:40.108 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:40.109 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:40.109 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:40.109 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:40.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0073340 con 0x7f63e0018390 2021-03-09T10:25:40.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.098+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:40.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.098+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:25:40.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.099+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042320 con 0x7f63e0018390 2021-03-09T10:25:40.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.470+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:40.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:40.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:40.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:40.583+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:41.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:41.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:41.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:41.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:42.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:42.583+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:42.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:42.583+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:43.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:43.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:43.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:43.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:44.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:44.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:44.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:44.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:45.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:25:45.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:45.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16065 cs=0 l=1 rev1=1 rx=0x7f63f00738e0 tx=0x7f63f0039e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:45.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16065 cs=0 l=1 rev1=1 rx=0x7f63f00738e0 tx=0x7f63f0039e20).stop 2021-03-09T10:25:45.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:45.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:45.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:45.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:45.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:25:45.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:45.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:45.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:45.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:45.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:45.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:45.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:45.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9d0 con 0x7f63d8003400 2021-03-09T10:25:45.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:45.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63d8009130 2021-03-09T10:25:45.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:45.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025c80 con 0x7f63d8003840 2021-03-09T10:25:45.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:45.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:45.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:45.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:45.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:45.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.096+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:45.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.096+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:25:45.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.096+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:45.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.096+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:45.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.096+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:45.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.096+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:45.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.096+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:45.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:45.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:45.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:45.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:45.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:45.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:25:45.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:25:45.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:45.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:45.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027c30 con 0x7f63d8003840 2021-03-09T10:25:45.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:45.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:45.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:25:45.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:25:45.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.097+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:45.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.098+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 secure :-1 s=READY pgs=16075 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc013f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:45.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.098+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:25:45.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.098+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:45.109 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:45.101+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44553 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:45.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63d8003840 2021-03-09T10:25:45.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.102+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:45.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.102+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:45.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.102+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:45.111 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:45.111 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:45.111 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:45.112 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:45.112 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:45.112 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:45.112 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:45.112 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:45.113 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:45.113 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:45.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc07aeb0 con 0x7f63d8003840 2021-03-09T10:25:45.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.105+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:45.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.105+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:25:45.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8003840 2021-03-09T10:25:45.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.471+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:45.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:45.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:46.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:46.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:46.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:46.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:47.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:47.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:47.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:47.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:48.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:48.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:48.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:48.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:49.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:49.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:49.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:49.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:50.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:50.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:50.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 secure :-1 s=READY pgs=16075 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc013f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:50.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 secure :-1 s=READY pgs=16075 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc013f30).stop 2021-03-09T10:25:50.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:50.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:50.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:50.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 unknown :-1 s=CLOSED pgs=16075 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:50.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:25:50.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:50.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:50.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:50.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:50.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:50.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:50.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:50.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:25:50.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:50.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021890 con 0x7f63d8009130 2021-03-09T10:25:50.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:50.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d830 con 0x7f63e0018390 2021-03-09T10:25:50.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:50.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:50.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:50.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:50.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:50.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:50.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:25:50.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:50.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:50.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:50.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.102+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:50.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:50.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:50.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:50.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:50.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:50.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:50.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:25:50.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:25:50.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:50.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:50.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c9f0 con 0x7f63e0018390 2021-03-09T10:25:50.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:50.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:50.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027c30 con 0x7f63e0018390 2021-03-09T10:25:50.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:25:50.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.103+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:50.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.104+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16088 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc012f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:50.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.104+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:25:50.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.104+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:50.117 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:50.107+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44592 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:50.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:25:50.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.108+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:50.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.108+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:50.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.108+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:50.118 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:50.118 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:50.119 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:50.119 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:50.119 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:50.119 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:50.119 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:50.120 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:50.120 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:50.120 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:50.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.111+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63e0018390 2021-03-09T10:25:50.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.111+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:50.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.111+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:25:50.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06f4a0 con 0x7f63e0018390 2021-03-09T10:25:50.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.472+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:50.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:50.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:50.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:50.585+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:51.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:51.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:51.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:51.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:52.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:52.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:52.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:52.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:53.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:53.584+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:53.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:53.584+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:54.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:54.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:54.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:54.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:25:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:25:55.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16088 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc012f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:25:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16088 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc012f40).stop 2021-03-09T10:25:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:25:55.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:25:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16088 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:25:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:25:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:25:55.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:25:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:25:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:25:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:55.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00147e0 con 0x7f63d8003400 2021-03-09T10:25:55.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:55.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:25:55.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:25:55.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079c40 con 0x7f63d8003840 2021-03-09T10:25:55.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:25:55.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:25:55.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:55.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:25:55.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.108+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:25:55.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.109+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:25:55.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.109+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:25:55.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.109+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:25:55.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.109+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:25:55.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.109+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:25:55.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.109+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:25:55.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.109+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:25:55.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.109+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:25:55.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:55.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:55.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:25:55.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:25:55.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:25:55.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:25:55.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:25:55.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:55.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:25:55.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:25:55.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:25:55.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024210 con 0x7f63d8003840 2021-03-09T10:25:55.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:25:55.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:55.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:25:55.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:25:55.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 secure :-1 s=READY pgs=16103 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004c260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:25:55.123 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:25:55.115+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44631 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:25:55.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0032f10 con 0x7f63d8003840 2021-03-09T10:25:55.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.116+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:25:55.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.116+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:25:55.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.116+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:25:55.124 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:25:55.125 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:25:55.125 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:55.125 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:25:55.125 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:25:55.125 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:25:55.126 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:25:55.126 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:25:55.126 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:25:55.126 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:25:55.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.120+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8003840 2021-03-09T10:25:55.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.120+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:25:55.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.120+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:25:55.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.120+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0039330 con 0x7f63d8003840 2021-03-09T10:25:55.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.473+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:25:55.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:55.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:55.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:56.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:56.585+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:56.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:56.585+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:57.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:57.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:57.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:57.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:58.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:58.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:58.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:58.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:25:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:59.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:25:59.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:25:59.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:00.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:00.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:00.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 secure :-1 s=READY pgs=16103 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004c260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:00.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 secure :-1 s=READY pgs=16103 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004c260).stop 2021-03-09T10:26:00.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:00.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:00.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:00.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021ae0 unknown :-1 s=CLOSED pgs=16103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:00.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:26:00.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:00.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:00.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:00.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:00.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:00.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:00.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:00.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025c80 con 0x7f63d8003400 2021-03-09T10:26:00.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:00.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63d8009130 2021-03-09T10:26:00.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:00.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021890 con 0x7f63e0018390 2021-03-09T10:26:00.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:00.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:00.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:00.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:00.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:00.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:00.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:26:00.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:00.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.116+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:00.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:00.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:00.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:00.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:00.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:00.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:00.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:00.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:00.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:26:00.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:26:00.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:00.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:00.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00216e0 con 0x7f63e0018390 2021-03-09T10:26:00.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:00.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:00.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c9f0 con 0x7f63e0018390 2021-03-09T10:26:00.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:26:00.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.117+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:00.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.118+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16117 cs=0 l=1 rev1=1 rx=0x7f63fc0237c0 tx=0x7f63fc023600).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:00.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.118+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:26:00.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.118+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:00.131 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:00.123+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44672 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:00.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.125+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:26:00.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.125+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:00.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.125+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:00.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.125+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:00.133 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:00.133 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:00.133 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:00.133 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:00.133 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:00.134 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:00.134 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:00.134 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:00.134 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:00.134 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:00.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051c60 con 0x7f63e0018390 2021-03-09T10:26:00.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.127+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:00.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.127+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:26:00.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:26:00.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.474+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:00.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:00.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:00.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:00.586+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:01.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:01.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:01.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:01.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:02.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:02.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:02.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:02.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:03.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:03.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:03.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:03.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:04.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:04.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:04.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:05.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:05.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:05.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16117 cs=0 l=1 rev1=1 rx=0x7f63fc0237c0 tx=0x7f63fc023600).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:05.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16117 cs=0 l=1 rev1=1 rx=0x7f63fc0237c0 tx=0x7f63fc023600).stop 2021-03-09T10:26:05.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:05.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:05.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:05.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:05.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:26:05.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:05.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:05.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:05.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:05.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:05.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:05.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:05.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029610 con 0x7f63d8003400 2021-03-09T10:26:05.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:05.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079c40 con 0x7f63d8009130 2021-03-09T10:26:05.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:05.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b180 con 0x7f63d8003840 2021-03-09T10:26:05.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:05.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:05.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.123+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:05.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.124+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:05.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.124+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:05.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.124+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:05.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.124+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:26:05.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.124+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:05.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.124+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.124+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.124+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.124+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:05.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:05.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:05.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:05.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:05.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:05.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:26:05.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:26:05.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:05.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:05.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001be20 con 0x7f63d8003840 2021-03-09T10:26:05.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:05.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:05.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:26:05.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:26:05.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.125+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:05.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.126+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:26:05.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.126+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:05.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012ad0 secure :-1 s=READY pgs=16132 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007f700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:05.139 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:05.129+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44709 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:05.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:26:05.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.130+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:05.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.130+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:05.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.130+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:05.141 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:05.141 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:05.141 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:05.141 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:05.142 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:05.142 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:05.142 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:05.142 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:05.143 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:05.143 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:05.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.136+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:26:05.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.136+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:05.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.136+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:26:05.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.136+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029de0 con 0x7f63d8003840 2021-03-09T10:26:05.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.475+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:05.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:05.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:05.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:06.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:06.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:06.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:06.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:07.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:07.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:07.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:07.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:08.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:08.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:08.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:08.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:09.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:09.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:09.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:09.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:10.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:10.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:10.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012ad0 secure :-1 s=READY pgs=16132 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007f700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:10.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012ad0 secure :-1 s=READY pgs=16132 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f007f700).stop 2021-03-09T10:26:10.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:10.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:10.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:10.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012ad0 unknown :-1 s=CLOSED pgs=16132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:10.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:26:10.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:10.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:10.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:10.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:10.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:10.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:10.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:10.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:26:10.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:10.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63d8009130 2021-03-09T10:26:10.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:10.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023c30 con 0x7f63e0018390 2021-03-09T10:26:10.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:10.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:10.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.130+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:10.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.131+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:10.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.131+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:10.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.131+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:10.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.131+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:26:10.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.131+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:10.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.131+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:10.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:10.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:10.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:10.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.132+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:10.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:10.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:26:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:26:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63e0018390 2021-03-09T10:26:10.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:10.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:10.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001be20 con 0x7f63e0018390 2021-03-09T10:26:10.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:26:10.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:10.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16154 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f007e620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:10.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:26:10.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.133+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:10.145 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:10.136+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44750 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:10.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:26:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.137+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.137+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.137+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:10.148 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:10.149 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:10.150 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:10.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.139+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:26:10.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.139+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:10.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.139+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:26:10.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.140+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63e0018390 2021-03-09T10:26:10.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.476+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:10.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.586+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:10.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.586+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:10.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:10.586+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:11.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:11.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:12.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:12.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:12.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:13.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:13.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:13.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:13.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:14.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:14.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:14.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:14.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:15.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:15.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:15.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16154 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f007e620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:15.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16154 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f007e620).stop 2021-03-09T10:26:15.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:15.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:15.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:15.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:15.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:26:15.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:15.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:15.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:15.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:15.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:15.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:15.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:15.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029610 con 0x7f63d8003400 2021-03-09T10:26:15.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:15.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:26:15.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:15.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cea0 con 0x7f63d8003840 2021-03-09T10:26:15.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:15.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:15.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:15.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:15.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:15.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.138+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:15.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.138+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:26:15.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.138+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:15.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.138+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:15.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.138+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:15.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.138+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:15.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.138+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:15.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:15.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:15.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:15.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:15.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:15.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:26:15.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:26:15.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:15.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:15.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00303f0 con 0x7f63d8003840 2021-03-09T10:26:15.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:15.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:15.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:26:15.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:26:15.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:15.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16164 cs=0 l=1 rev1=1 rx=0x7f63f007e5b0 tx=0x7f63f0013bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:15.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:26:15.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.139+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:15.151 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:15.142+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44785 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:15.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:26:15.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.143+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:15.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.143+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:15.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.143+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:15.153 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:15.153 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:15.153 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:15.153 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:15.153 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:15.154 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:15.154 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:15.154 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:15.154 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:15.155 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:15.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.145+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:26:15.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.145+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:15.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.145+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:26:15.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:26:15.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.477+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:15.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:15.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:15.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:16.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:16.588+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:16.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:16.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:17.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:17.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:17.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:17.587+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:18.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:18.587+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:18.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:18.588+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:19.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:19.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:19.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:20.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:20.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:20.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16164 cs=0 l=1 rev1=1 rx=0x7f63f007e5b0 tx=0x7f63f0013bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:20.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16164 cs=0 l=1 rev1=1 rx=0x7f63f007e5b0 tx=0x7f63f0013bc0).stop 2021-03-09T10:26:20.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:20.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:20.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:20.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=16164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:20.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:26:20.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:20.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:20.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:20.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:20.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:20.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:20.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:20.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000cae0 con 0x7f63d8003400 2021-03-09T10:26:20.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:20.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b180 con 0x7f63d8009130 2021-03-09T10:26:20.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:20.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021890 con 0x7f63e0018390 2021-03-09T10:26:20.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:20.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:20.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:20.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:20.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.143+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:20.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.144+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:20.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.144+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:26:20.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.144+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:20.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.144+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:20.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:20.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:20.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:20.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:20.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:20.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:20.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:20.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:20.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:26:20.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:26:20.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:20.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:20.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c9f0 con 0x7f63e0018390 2021-03-09T10:26:20.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:20.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:20.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00216e0 con 0x7f63e0018390 2021-03-09T10:26:20.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:26:20.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.145+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:20.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.146+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:26:20.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.146+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16180 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:20.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.146+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:20.158 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:20.149+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44818 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:20.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:26:20.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.150+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:20.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.150+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:20.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.150+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:20.159 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:20.160 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:20.160 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:20.160 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:20.160 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:20.160 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:20.161 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:20.161 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:20.161 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:20.161 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:20.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065c10 con 0x7f63e0018390 2021-03-09T10:26:20.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.155+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:20.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.155+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:26:20.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:26:20.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.478+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:20.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:20.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:20.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:20.589+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:21.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:21.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:21.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:21.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:22.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:22.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:22.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:23.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:23.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:23.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:23.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:24.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:24.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:24.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:24.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:25.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:25.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:25.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16180 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:25.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16180 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc009990).stop 2021-03-09T10:26:25.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:25.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:25.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:25.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:25.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:26:25.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:25.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:25.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:25.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:25.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:25.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:25.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:25.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63d8003400 2021-03-09T10:26:25.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:25.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00245c0 con 0x7f63d8009130 2021-03-09T10:26:25.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:25.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021890 con 0x7f63d8003840 2021-03-09T10:26:25.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:25.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:25.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:25.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:25.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:25.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:25.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T10:26:25.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:26:25.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:25.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:25.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2021-03-09T10:26:25.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2021-03-09T10:26:25.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.151+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:25.161 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:25.156+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44863 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:25.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:25.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T10:26:25.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:25.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T10:26:25.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:25.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:26:25.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:25.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:25.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:25.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:25.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:25.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:25.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:25.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:25.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:25.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:25.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:26:25.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:26:25.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:25.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:25.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066120 con 0x7f63d8003840 2021-03-09T10:26:25.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:25.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:25.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00303f0 con 0x7f63d8003840 2021-03-09T10:26:25.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:26:25.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:25.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:26:25.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:25.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16199 cs=0 l=1 rev1=1 rx=0x7f63f0031490 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:25.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:26:25.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:25.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:25.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:25.363 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:25.363 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:25.363 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:25.364 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:25.364 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:25.364 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:25.364 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:25.364 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:25.365 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:25.365 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:25.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:26:25.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:25.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.352+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:26:25.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.353+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:26:25.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.479+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:25.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:25.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:25.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:26.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:26.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:26.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:26.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:27.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:27.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:27.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:27.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:28.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:28.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:28.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:28.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:29.589+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:29.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:29.589+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:30.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:26:30.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:30.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16199 cs=0 l=1 rev1=1 rx=0x7f63f0031490 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:30.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16199 cs=0 l=1 rev1=1 rx=0x7f63f0031490 tx=0x7f63f0006610).stop 2021-03-09T10:26:30.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:30.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:30.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:30.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=16199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:30.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:26:30.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:30.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:30.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:30.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:30.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:30.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:30.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:30.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:26:30.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:30.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63d8009130 2021-03-09T10:26:30.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:30.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00245c0 con 0x7f63e0018390 2021-03-09T10:26:30.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:30.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:30.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:30.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:30.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.158+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:30.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.159+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:30.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.159+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:26:30.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.159+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:30.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.159+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:30.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:30.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:30.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:30.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:30.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:30.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:30.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:30.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:30.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:26:30.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:26:30.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:30.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:30.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024210 con 0x7f63e0018390 2021-03-09T10:26:30.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:30.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:30.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066120 con 0x7f63e0018390 2021-03-09T10:26:30.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:26:30.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:30.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16207 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0041f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:30.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:26:30.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.160+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:30.173 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:30.163+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44906 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:30.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0031500 con 0x7f63e0018390 2021-03-09T10:26:30.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.164+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:30.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.164+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:30.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.164+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:30.174 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:30.175 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:30.175 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:30.175 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:30.175 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:30.176 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:30.176 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:30.176 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:30.176 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:30.176 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:30.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63e0018390 2021-03-09T10:26:30.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.167+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:30.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.167+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:26:30.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00312f0 con 0x7f63e0018390 2021-03-09T10:26:30.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.480+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:30.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:30.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:30.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:30.590+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:31.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:31.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:31.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:31.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:32.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:32.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:32.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:32.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:33.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:33.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:33.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:33.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:34.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:34.590+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:34.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:34.590+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:35.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:35.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:35.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16207 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0041f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:35.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16207 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0041f00).stop 2021-03-09T10:26:35.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:35.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:35.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:35.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:35.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:26:35.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:35.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:35.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:35.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:35.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:35.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:35.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:35.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021890 con 0x7f63d8003400 2021-03-09T10:26:35.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:35.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63d8009130 2021-03-09T10:26:35.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:35.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ce40 con 0x7f63d8003840 2021-03-09T10:26:35.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:35.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:35.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.164+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:35.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.165+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:35.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.165+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:35.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.165+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:35.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.165+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:26:35.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.165+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:35.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.165+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:35.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.165+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:35.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.165+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:35.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.165+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:35.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:35.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:35.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:35.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:35.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:35.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:26:35.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:26:35.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:35.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:35.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:26:35.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:35.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:35.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024210 con 0x7f63d8003840 2021-03-09T10:26:35.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:26:35.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.167+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:35.179 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:35.171+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44943 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:35.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.171+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:26:35.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.171+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16218 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f00361e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:35.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.171+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:35.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00483d0 con 0x7f63d8003840 2021-03-09T10:26:35.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.173+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:35.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.173+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:35.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.173+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:35.181 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:35.181 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:35.182 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:35.182 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:35.182 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:35.182 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:35.182 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:35.183 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:35.183 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:35.183 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:35.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.175+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00481f0 con 0x7f63d8003840 2021-03-09T10:26:35.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.175+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:35.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.175+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:26:35.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.176+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:26:35.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.481+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:35.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:35.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:35.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:36.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:36.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:36.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:36.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:37.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:37.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:37.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:37.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:38.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:38.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:38.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:38.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:39.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:39.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:39.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:39.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:26:40.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:40.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16218 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f00361e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:40.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16218 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f00361e0).stop 2021-03-09T10:26:40.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:40.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:40.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:40.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=16218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:40.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:26:40.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:40.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:40.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:40.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:40.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:40.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:40.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:40.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:26:40.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:40.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:26:40.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:40.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63e0018390 2021-03-09T10:26:40.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:40.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:40.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:40.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:40.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:40.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.173+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:40.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.173+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:26:40.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.173+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:40.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.173+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:40.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:40.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:40.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:40.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:40.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:40.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:40.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:40.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:40.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:26:40.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:26:40.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.174+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:40.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:40.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00162e0 con 0x7f63e0018390 2021-03-09T10:26:40.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:40.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:40.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c9f0 con 0x7f63e0018390 2021-03-09T10:26:40.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:26:40.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:40.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16233 cs=0 l=1 rev1=1 rx=0x7f63fc012ff0 tx=0x7f63fc023260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:40.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:26:40.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.175+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:40.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065c10 con 0x7f63e0018390 2021-03-09T10:26:40.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.179+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:40.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.179+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:40.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.179+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:40.188 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:40.188 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:40.188 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:40.188 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:40.189 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:40.189 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:40.189 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:40.189 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:40.190 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:40.190 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:40.190 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:40.179+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 44984 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:40.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.182+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065c10 con 0x7f63e0018390 2021-03-09T10:26:40.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.182+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:40.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.182+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:26:40.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065d80 con 0x7f63e0018390 2021-03-09T10:26:40.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.482+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:40.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:40.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:40.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:40.591+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:41.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:41.591+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:41.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:41.591+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:42.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:42.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:42.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:42.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:43.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:43.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:43.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:43.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:44.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:44.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:44.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:44.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:45.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:45.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:45.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16233 cs=0 l=1 rev1=1 rx=0x7f63fc012ff0 tx=0x7f63fc023260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:45.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16233 cs=0 l=1 rev1=1 rx=0x7f63fc012ff0 tx=0x7f63fc023260).stop 2021-03-09T10:26:45.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:45.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:45.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:45.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:45.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:26:45.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:45.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:45.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021890 con 0x7f63d8003400 2021-03-09T10:26:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:26:45.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ddf0 con 0x7f63d8003840 2021-03-09T10:26:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.180+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.180+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:45.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.180+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:45.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.180+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:45.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.180+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:45.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.180+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:45.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:26:45.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:45.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:45.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:45.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:45.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:45.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:45.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:45.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:45.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:45.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:45.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:26:45.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:26:45.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:45.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:45.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:26:45.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.181+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:26:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.182+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:26:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.182+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:45.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.182+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16245 cs=0 l=1 rev1=1 rx=0x7f63f005dde0 tx=0x7f63f003ed60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:45.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.182+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:26:45.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.182+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:45.194 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:45.187+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45019 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:45.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:26:45.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.188+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:45.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.188+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:45.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.188+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:45.196 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:45.197 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:45.197 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:45.197 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:45.197 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:45.197 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:45.198 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:45.198 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:45.198 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:45.198 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:45.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:26:45.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.188+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:45.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.188+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:26:45.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.189+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:26:45.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.483+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:45.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:45.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:46.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:46.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:46.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:46.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:47.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:47.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:47.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:47.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:48.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:48.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:48.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:48.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:49.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:49.592+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:49.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:49.592+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:50.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:50.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:50.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16245 cs=0 l=1 rev1=1 rx=0x7f63f005dde0 tx=0x7f63f003ed60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:50.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16245 cs=0 l=1 rev1=1 rx=0x7f63f005dde0 tx=0x7f63f003ed60).stop 2021-03-09T10:26:50.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:50.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:50.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=16245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:26:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:50.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:50.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:50.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c160 con 0x7f63d8003400 2021-03-09T10:26:50.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:50.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:26:50.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:50.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63e0018390 2021-03-09T10:26:50.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.187+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:50.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.188+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:50.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.188+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:50.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.188+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:50.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.188+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.189+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:50.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.189+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:26:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.189+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.189+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:50.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:50.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:50.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:50.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:26:50.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:26:50.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:50.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:50.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00178a0 con 0x7f63e0018390 2021-03-09T10:26:50.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:50.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:50.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00162e0 con 0x7f63e0018390 2021-03-09T10:26:50.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:26:50.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.190+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:50.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.191+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16260 cs=0 l=1 rev1=1 rx=0x7f63e00356f0 tx=0x7f63fc052f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:50.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.191+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:26:50.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.191+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:50.202 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:50.195+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45062 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:50.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058650 con 0x7f63e0018390 2021-03-09T10:26:50.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.195+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:50.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.195+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:50.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.195+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:50.204 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:50.204 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:50.204 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:50.204 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:50.205 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:50.205 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:50.205 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:50.205 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:50.206 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:50.206 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:50.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058650 con 0x7f63e0018390 2021-03-09T10:26:50.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.196+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:50.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.196+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:26:50.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.197+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63e0018390 2021-03-09T10:26:50.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.484+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:50.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:50.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:50.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:50.593+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:51.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:51.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:51.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:51.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:52.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:52.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:52.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:52.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:53.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:53.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:53.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:53.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:54.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:54.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:54.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:54.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:55.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:26:55.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:26:55.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16260 cs=0 l=1 rev1=1 rx=0x7f63e00356f0 tx=0x7f63fc052f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:26:55.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16260 cs=0 l=1 rev1=1 rx=0x7f63e00356f0 tx=0x7f63fc052f70).stop 2021-03-09T10:26:55.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:55.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:26:55.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:26:55.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:26:55.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:26:55.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:55.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:26:55.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:55.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:26:55.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:26:55.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:26:55.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:55.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:26:55.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:55.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ddf0 con 0x7f63d8009130 2021-03-09T10:26:55.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:26:55.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00147e0 con 0x7f63d8003840 2021-03-09T10:26:55.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:26:55.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:26:55.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.195+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:55.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:26:55.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.196+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:26:55.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.196+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:26:55.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.196+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:26:55.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.196+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:26:55.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.196+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:26:55.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.197+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:26:55.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.197+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:26:55.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.197+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:26:55.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:26:55.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:55.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:55.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:26:55.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:26:55.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:26:55.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:26:55.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:26:55.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:55.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027c30 con 0x7f63d8003840 2021-03-09T10:26:55.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:26:55.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:26:55.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cc10 con 0x7f63d8003840 2021-03-09T10:26:55.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:26:55.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:55.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16272 cs=0 l=1 rev1=1 rx=0x7f63f0000cc0 tx=0x7f63f0055370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:26:55.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:26:55.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.198+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:26:55.210 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:26:55.202+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45097 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:26:55.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.203+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:26:55.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.203+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:26:55.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.203+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:26:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.203+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:26:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:26:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:26:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:55.212 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:26:55.213 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:26:55.213 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:26:55.213 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:26:55.213 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:26:55.214 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:26:55.214 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:26:55.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:26:55.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.206+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:26:55.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.206+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:26:55.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:26:55.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.485+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:26:55.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:55.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:55.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:56.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:56.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:56.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:56.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:57.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:57.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:57.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:57.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:58.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:58.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:58.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:58.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:26:59.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:59.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:26:59.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:26:59.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:00.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:00.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:00.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16272 cs=0 l=1 rev1=1 rx=0x7f63f0000cc0 tx=0x7f63f0055370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:00.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16272 cs=0 l=1 rev1=1 rx=0x7f63f0000cc0 tx=0x7f63f0055370).stop 2021-03-09T10:27:00.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:00.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:00.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:00.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=16272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:00.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:27:00.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:00.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:00.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:00.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.202+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:00.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:00.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:00.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:00.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028bc0 con 0x7f63d8003400 2021-03-09T10:27:00.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:00.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001890 con 0x7f63d8009130 2021-03-09T10:27:00.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:00.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63e0018390 2021-03-09T10:27:00.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:00.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:00.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:00.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:00.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:27:00.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:00.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:00.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:00.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.203+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:00.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.204+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:00.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.204+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:00.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.204+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:00.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:00.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:00.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:00.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:00.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:00.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:27:00.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:27:00.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:00.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:00.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00102a0 con 0x7f63e0018390 2021-03-09T10:27:00.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:00.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:00.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00178a0 con 0x7f63e0018390 2021-03-09T10:27:00.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:27:00.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.205+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:00.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.206+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16289 cs=0 l=1 rev1=1 rx=0x7f63fc04bc00 tx=0x7f63fc04bb20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:00.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.206+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:27:00.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.206+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:00.216 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:00.208+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45140 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:00.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:27:00.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.209+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:00.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.209+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:00.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.209+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:00.218 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:00.218 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:00.219 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:00.219 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:00.219 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:00.219 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:00.220 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:00.220 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:00.220 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:00.220 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:00.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.212+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:27:00.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.212+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:00.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.212+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:27:00.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:27:00.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.486+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:00.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:00.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:00.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:00.594+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:01.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:01.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:01.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:01.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:02.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:02.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:02.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:02.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:03.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:03.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:03.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:03.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:04.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:04.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:04.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:04.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:05.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:05.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:05.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16289 cs=0 l=1 rev1=1 rx=0x7f63fc04bc00 tx=0x7f63fc04bb20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:05.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16289 cs=0 l=1 rev1=1 rx=0x7f63fc04bc00 tx=0x7f63fc04bb20).stop 2021-03-09T10:27:05.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:05.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:05.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:05.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:05.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:27:05.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:05.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:05.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:05.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:05.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:05.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:05.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:05.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021890 con 0x7f63d8003400 2021-03-09T10:27:05.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:27:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8003840 2021-03-09T10:27:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.208+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:05.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024e10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:27:05.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:05.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:05.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.209+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:05.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024e10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:27:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:27:05.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028770 con 0x7f63d8003840 2021-03-09T10:27:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:05.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027c30 con 0x7f63d8003840 2021-03-09T10:27:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:27:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.210+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.211+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:27:05.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.211+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ccc0 secure :-1 s=READY pgs=16298 cs=0 l=1 rev1=1 rx=0x7f63f0003580 tx=0x7f63f0006c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:05.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.211+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:05.223 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:05.214+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45177 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.214+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63d8003840 2021-03-09T10:27:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.214+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:05.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.214+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.214+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:05.225 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:05.226 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:05.227 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:05.227 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:05.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.214+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63d8003840 2021-03-09T10:27:05.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.214+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:05.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.214+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:27:05.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028d70 con 0x7f63d8003840 2021-03-09T10:27:05.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.486+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:05.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.593+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:05.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:05.593+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:06.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:06.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:06.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:06.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:07.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:07.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:07.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:07.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:08.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:08.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:08.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:08.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:09.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:09.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:09.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.214+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:10.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.214+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:10.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.214+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ccc0 secure :-1 s=READY pgs=16298 cs=0 l=1 rev1=1 rx=0x7f63f0003580 tx=0x7f63f0006c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:10.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ccc0 secure :-1 s=READY pgs=16298 cs=0 l=1 rev1=1 rx=0x7f63f0003580 tx=0x7f63f0006c30).stop 2021-03-09T10:27:10.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:10.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000ccc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000ccc0 unknown :-1 s=CLOSED pgs=16298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:27:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:10.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:10.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:10.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:10.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:10.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:10.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:10.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00116d0 con 0x7f63d8003400 2021-03-09T10:27:10.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:10.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b990 con 0x7f63d8009130 2021-03-09T10:27:10.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:10.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63e0018390 2021-03-09T10:27:10.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:10.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:10.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:10.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:10.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:10.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:10.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:27:10.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:10.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.215+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:10.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:10.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:10.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:10.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:10.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:10.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:27:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:27:10.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:10.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:10.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:27:10.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:10.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:10.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00102a0 con 0x7f63e0018390 2021-03-09T10:27:10.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.217+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:27:10.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.217+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:10.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.217+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:27:10.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.217+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16312 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:10.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.217+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:10.230 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:10.221+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45218 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:27:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.222+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:10.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.222+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.222+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:10.232 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:10.233 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:10.234 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:27:10.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.222+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:10.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.222+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:27:10.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:27:10.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.487+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:10.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:10.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:10.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:10.594+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:11.594+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:11.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:11.594+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:12.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:12.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:12.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:12.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:13.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:13.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:13.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:13.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:14.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:14.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:14.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:14.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:15.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:15.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:15.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16312 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:15.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16312 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc009a00).stop 2021-03-09T10:27:15.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:15.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:15.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:15.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:15.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:27:15.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:15.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:15.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:15.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:15.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:15.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:15.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:15.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b860 con 0x7f63d8003400 2021-03-09T10:27:15.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:15.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:27:15.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:15.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025120 con 0x7f63d8003840 2021-03-09T10:27:15.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:15.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:15.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:15.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:15.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:15.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:15.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:27:15.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:15.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.222+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:15.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.223+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:15.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.223+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:15.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.223+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:15.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:15.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:15.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:15.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:15.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:15.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:27:15.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:27:15.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:15.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:15.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007ad90 con 0x7f63d8003840 2021-03-09T10:27:15.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:15.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:15.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028770 con 0x7f63d8003840 2021-03-09T10:27:15.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:27:15.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.224+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:15.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.225+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=16326 cs=0 l=1 rev1=1 rx=0x7f63f0027820 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:15.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.225+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:27:15.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.225+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:15.238 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:15.228+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45255 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:15.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:27:15.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.229+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:15.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.229+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:15.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.229+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:15.240 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:15.240 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:15.240 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:15.241 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:15.241 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:15.241 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:15.241 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:15.241 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:15.242 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:15.242 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:15.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:27:15.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.232+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:15.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.232+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:27:15.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.233+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:27:15.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.488+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:15.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:15.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:15.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:16.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:16.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:16.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:16.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:17.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:17.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:17.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:17.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:18.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:18.595+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:18.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:18.595+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:19.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:19.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:19.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:19.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:20.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=16326 cs=0 l=1 rev1=1 rx=0x7f63f0027820 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:20.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=16326 cs=0 l=1 rev1=1 rx=0x7f63f0027820 tx=0x7f63f004db20).stop 2021-03-09T10:27:20.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=CLOSED pgs=16326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:27:20.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:20.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068d00 con 0x7f63d8003400 2021-03-09T10:27:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:20.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8009130 2021-03-09T10:27:20.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:20.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63e0018390 2021-03-09T10:27:20.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:20.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:20.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:20.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:20.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.229+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:20.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.230+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:20.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.230+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:27:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.230+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.230+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:20.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:20.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:27:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:27:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:20.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:20.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00655e0 con 0x7f63e0018390 2021-03-09T10:27:20.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:20.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:20.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007ad90 con 0x7f63e0018390 2021-03-09T10:27:20.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:27:20.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.231+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:20.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16341 cs=0 l=1 rev1=1 rx=0x7f63f0006650 tx=0x7f63f007a330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:20.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.232+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:27:20.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.232+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:20.293 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:20.235+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45296 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:20.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63e0018390 2021-03-09T10:27:20.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.237+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:20.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.237+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:20.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.237+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:20.294 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:20.295 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:20.295 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:20.295 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:20.295 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:20.295 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:20.296 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:20.296 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:20.296 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:20.296 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:20.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.239+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63e0018390 2021-03-09T10:27:20.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.239+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:20.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.239+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:27:20.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.240+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63e0018390 2021-03-09T10:27:20.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.489+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:20.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:20.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:20.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:20.596+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:21.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:21.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:21.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:21.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:22.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:22.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:22.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:23.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:23.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:23.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:24.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:24.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:24.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:24.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:25.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:27:25.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:25.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16341 cs=0 l=1 rev1=1 rx=0x7f63f0006650 tx=0x7f63f007a330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:25.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16341 cs=0 l=1 rev1=1 rx=0x7f63f0006650 tx=0x7f63f007a330).stop 2021-03-09T10:27:25.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:25.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:25.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:25.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:25.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:27:25.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:25.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:25.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:25.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:25.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:25.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:25.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:25.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cdd0 con 0x7f63d8003400 2021-03-09T10:27:25.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:25.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:27:25.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:25.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac10 con 0x7f63d8003840 2021-03-09T10:27:25.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:25.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:25.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:25.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:25.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:25.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:25.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:27:25.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:25.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:25.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:25.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:25.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:25.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:25.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:25.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.237+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:25.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:25.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:25.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:27:25.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:27:25.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:25.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:25.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027c30 con 0x7f63d8003840 2021-03-09T10:27:25.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:25.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:25.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00655e0 con 0x7f63d8003840 2021-03-09T10:27:25.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:27:25.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:25.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:27:25.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:25.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.238+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16353 cs=0 l=1 rev1=1 rx=0x7f63f0074020 tx=0x7f63f0029510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:25.253 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:25.242+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45333 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:25.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.242+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0029ee0 con 0x7f63d8003840 2021-03-09T10:27:25.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.242+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:25.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.242+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:25.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.242+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:25.255 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:25.255 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:25.255 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:25.255 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:25.256 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:25.256 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:25.256 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:25.256 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:25.256 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:25.257 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:25.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.242+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005fe90 con 0x7f63d8003840 2021-03-09T10:27:25.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.242+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:25.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.242+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:27:25.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:27:25.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.490+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:25.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:25.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:25.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:26.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:26.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:26.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:26.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:27.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:27.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:27.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:27.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:28.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:28.596+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:28.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:28.596+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:29.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:29.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:29.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:29.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:30.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.242+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:30.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:30.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16353 cs=0 l=1 rev1=1 rx=0x7f63f0074020 tx=0x7f63f0029510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:30.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16353 cs=0 l=1 rev1=1 rx=0x7f63f0074020 tx=0x7f63f0029510).stop 2021-03-09T10:27:30.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:30.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:30.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:30.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=16353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:30.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:27:30.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:30.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:30.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:30.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:30.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:30.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:30.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:30.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009670 con 0x7f63d8003400 2021-03-09T10:27:30.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:30.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8009130 2021-03-09T10:27:30.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:30.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63e0018390 2021-03-09T10:27:30.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:30.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:30.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:30.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:30.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:30.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:30.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:27:30.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:30.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:30.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:30.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:30.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:30.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:30.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:30.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:30.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:30.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:30.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:27:30.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:27:30.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:30.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:30.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bf00 con 0x7f63e0018390 2021-03-09T10:27:30.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:30.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:30.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027c30 con 0x7f63e0018390 2021-03-09T10:27:30.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:27:30.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.244+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:30.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.245+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16367 cs=0 l=1 rev1=1 rx=0x7f63f007f6d0 tx=0x7f63f007b490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:30.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.245+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:27:30.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.245+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:30.258 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:30.249+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45374 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:30.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63e0018390 2021-03-09T10:27:30.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.249+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:30.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.249+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:30.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.249+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:30.260 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:30.260 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:30.261 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:30.261 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:30.261 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:30.261 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:30.261 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:30.262 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:30.262 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:30.262 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:30.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028470 con 0x7f63e0018390 2021-03-09T10:27:30.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.249+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:30.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.249+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:27:30.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0080350 con 0x7f63e0018390 2021-03-09T10:27:30.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.491+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:30.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:30.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:30.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:30.597+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:31.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:31.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:31.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:31.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:32.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:32.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:33.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:33.597+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:33.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:33.597+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:34.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:34.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:34.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:34.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:35.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:35.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:35.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16367 cs=0 l=1 rev1=1 rx=0x7f63f007f6d0 tx=0x7f63f007b490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:35.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16367 cs=0 l=1 rev1=1 rx=0x7f63f007f6d0 tx=0x7f63f007b490).stop 2021-03-09T10:27:35.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:35.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:35.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:35.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:35.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:27:35.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:35.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:35.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:35.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:35.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:35.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:35.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:35.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:27:35.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:35.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025be0 con 0x7f63d8009130 2021-03-09T10:27:35.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:35.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac10 con 0x7f63d8003840 2021-03-09T10:27:35.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.249+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:35.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:35.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:35.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:35.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:35.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:35.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:27:35.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:35.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:35.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:35.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:35.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.250+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:35.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:35.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:35.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:35.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:35.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:35.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:27:35.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:27:35.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:35.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:35.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002cb0 con 0x7f63d8003840 2021-03-09T10:27:35.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:35.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:35.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bf00 con 0x7f63d8003840 2021-03-09T10:27:35.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:27:35.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:35.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16378 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f007b500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:35.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:27:35.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.251+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:35.266 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:35.256+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45411 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:35.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.256+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0012350 con 0x7f63d8003840 2021-03-09T10:27:35.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.256+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:35.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.256+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:35.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.256+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:35.268 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:35.268 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:35.268 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:35.269 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:35.269 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:35.269 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:35.269 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:35.270 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:35.270 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:35.270 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:35.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.256+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:27:35.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.256+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:35.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.256+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:27:35.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029b70 con 0x7f63d8003840 2021-03-09T10:27:35.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.492+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:35.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:35.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:35.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:36.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:36.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:36.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:36.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:37.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:37.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:37.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:37.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:38.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:38.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:38.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:38.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:39.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:39.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:39.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:39.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:40.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:40.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:40.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16378 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f007b500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:40.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16378 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f007b500).stop 2021-03-09T10:27:40.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:40.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:40.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:40.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=16378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:40.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:27:40.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:40.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:40.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:40.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:40.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:40.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:40.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:40.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8003400 2021-03-09T10:27:40.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:40.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0002b70 con 0x7f63d8009130 2021-03-09T10:27:40.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:40.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e0018390 2021-03-09T10:27:40.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:40.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.256+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:40.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:40.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:40.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:40.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:40.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:27:40.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:40.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:40.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:40.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:40.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.257+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:40.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:40.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:40.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:40.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:40.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:40.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:27:40.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:27:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079bf0 con 0x7f63e0018390 2021-03-09T10:27:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:40.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:27:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:27:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.258+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:40.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.259+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16395 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc0715c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.259+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:27:40.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.259+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:40.271 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:40.263+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45452 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:40.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:27:40.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.265+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:40.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.265+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:40.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.265+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:40.273 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:40.273 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:40.273 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:40.273 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:40.274 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:40.275 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065e40 con 0x7f63e0018390 2021-03-09T10:27:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.265+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:40.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.265+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:27:40.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:27:40.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.493+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:40.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:40.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:40.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:40.598+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:41.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:41.598+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:41.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:41.598+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:42.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:42.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:42.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:42.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:43.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:43.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:43.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:43.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:44.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:44.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:44.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:45.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:45.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:45.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16395 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc0715c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:45.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16395 cs=0 l=1 rev1=1 rx=0x7f63fc006520 tx=0x7f63fc0715c0).stop 2021-03-09T10:27:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:45.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:27:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:45.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:45.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:45.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:45.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:45.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:27:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001dc0 con 0x7f63d8009130 2021-03-09T10:27:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:27:45.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:27:45.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:45.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:45.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:45.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.264+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:45.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:45.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:45.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:45.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:45.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:27:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:27:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:45.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00162e0 con 0x7f63d8003840 2021-03-09T10:27:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002cb0 con 0x7f63d8003840 2021-03-09T10:27:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:27:45.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.265+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:45.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.266+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=16406 cs=0 l=1 rev1=1 rx=0x7f63f0080ff0 tx=0x7f63f0003570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:45.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.266+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:27:45.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.266+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:45.280 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:45.272+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45489 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:45.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.272+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63d8003840 2021-03-09T10:27:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.272+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.272+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.272+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:45.281 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:45.282 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:45.282 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:45.282 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:45.282 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:45.282 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:45.283 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:45.284 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.272+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63d8003840 2021-03-09T10:27:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.272+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.272+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:27:45.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003e180 con 0x7f63d8003840 2021-03-09T10:27:45.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.494+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:45.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:46.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:46.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:46.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:46.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:47.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:47.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:47.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:47.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:48.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:48.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:48.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:48.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:49.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:49.599+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:49.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:49.599+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:50.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:27:50.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:50.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=16406 cs=0 l=1 rev1=1 rx=0x7f63f0080ff0 tx=0x7f63f0003570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:50.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=16406 cs=0 l=1 rev1=1 rx=0x7f63f0080ff0 tx=0x7f63f0003570).stop 2021-03-09T10:27:50.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:50.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:50.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:50.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=CLOSED pgs=16406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:50.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:27:50.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:50.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:50.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:50.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:50.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:50.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:50.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:50.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac10 con 0x7f63d8003400 2021-03-09T10:27:50.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:50.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a2f0 con 0x7f63d8009130 2021-03-09T10:27:50.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:50.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b550 con 0x7f63e0018390 2021-03-09T10:27:50.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:50.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:50.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:50.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:50.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.272+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:50.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.273+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:50.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.273+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:27:50.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.273+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:50.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.273+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:50.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.273+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:50.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.273+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:50.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.273+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:50.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:50.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:50.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:50.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:50.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:50.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:27:50.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:27:50.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:50.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:50.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007a160 con 0x7f63e0018390 2021-03-09T10:27:50.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:50.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:50.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00162e0 con 0x7f63e0018390 2021-03-09T10:27:50.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:27:50.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.274+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:50.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.275+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:27:50.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.275+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:50.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.275+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16421 cs=0 l=1 rev1=1 rx=0x7f63f00035e0 tx=0x7f63f007f700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:50.287 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:50.278+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45530 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:50.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028e90 con 0x7f63e0018390 2021-03-09T10:27:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.279+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.279+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.279+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:50.289 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:50.290 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:50.290 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:50.290 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:50.290 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:50.291 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:50.291 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:50.291 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:50.291 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:50.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.283+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63e0018390 2021-03-09T10:27:50.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.283+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:50.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.283+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:27:50.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63e0018390 2021-03-09T10:27:50.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.495+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:50.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:50.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:50.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:50.600+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:51.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:51.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:51.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:51.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:52.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:52.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:52.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:52.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:53.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:53.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:53.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:53.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:54.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:54.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:54.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:54.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:55.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:27:55.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:27:55.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16421 cs=0 l=1 rev1=1 rx=0x7f63f00035e0 tx=0x7f63f007f700).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:27:55.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16421 cs=0 l=1 rev1=1 rx=0x7f63f00035e0 tx=0x7f63f007f700).stop 2021-03-09T10:27:55.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:55.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:27:55.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:27:55.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:27:55.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:27:55.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0019140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:55.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:27:55.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:55.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:27:55.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:27:55.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:27:55.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:55.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:27:55.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:55.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005eb70 con 0x7f63d8009130 2021-03-09T10:27:55.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:27:55.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022260 con 0x7f63d8003840 2021-03-09T10:27:55.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:27:55.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:27:55.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:55.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:27:55.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:27:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0019140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:27:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:27:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:27:55.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.279+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:27:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.280+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:27:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.280+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:27:55.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.280+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:27:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:27:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:27:55.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:27:55.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:27:55.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:27:55.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:27:55.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:55.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d550 con 0x7f63d8003840 2021-03-09T10:27:55.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:27:55.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:27:55.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007a160 con 0x7f63d8003840 2021-03-09T10:27:55.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:27:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.281+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.282+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:27:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.282+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:27:55.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.282+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0019140 secure :-1 s=READY pgs=16438 cs=0 l=1 rev1=1 rx=0x7f63e00290f0 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:27:55.294 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:27:55.286+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45567 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:27:55.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.287+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0059ed0 con 0x7f63d8003840 2021-03-09T10:27:55.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.287+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:27:55.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.287+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:27:55.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.287+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:27:55.295 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:27:55.296 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:27:55.296 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:55.296 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:27:55.296 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:27:55.296 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:27:55.297 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:27:55.297 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:27:55.297 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:27:55.297 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:27:55.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.287+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0032ee0 con 0x7f63d8003840 2021-03-09T10:27:55.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.287+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:27:55.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.287+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:27:55.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.288+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0088d10 con 0x7f63d8003840 2021-03-09T10:27:55.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.496+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:27:55.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:55.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:55.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:56.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:56.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:56.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:56.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:57.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:57.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:57.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:57.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:58.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:58.600+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:58.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:58.600+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:27:59.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:59.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:27:59.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:27:59.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:00.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0019140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:00.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0019140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:00.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0019140 secure :-1 s=READY pgs=16438 cs=0 l=1 rev1=1 rx=0x7f63e00290f0 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:00.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0019140 secure :-1 s=READY pgs=16438 cs=0 l=1 rev1=1 rx=0x7f63e00290f0 tx=0x7f63f004de00).stop 2021-03-09T10:28:00.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:00.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:00.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0019140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:00.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0019140 unknown :-1 s=CLOSED pgs=16438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:00.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:28:00.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:00.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:00.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:00.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:00.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:00.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:00.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:00.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:28:00.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:00.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:28:00.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:00.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001dc0 con 0x7f63e0018390 2021-03-09T10:28:00.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:00.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.286+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:00.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:00.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:00.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:00.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:00.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:28:00.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:00.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:00.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:00.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:00.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:00.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:00.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:00.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:00.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:00.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:00.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:28:00.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:28:00.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.287+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:00.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.288+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:00.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00162e0 con 0x7f63e0018390 2021-03-09T10:28:00.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.288+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:00.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.288+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:00.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.288+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d550 con 0x7f63e0018390 2021-03-09T10:28:00.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.288+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:28:00.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.288+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:00.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.289+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16448 cs=0 l=1 rev1=1 rx=0x7f63f007af40 tx=0x7f63f0027ee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:00.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.289+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:28:00.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.289+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:00.302 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:00.293+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45608 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:00.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.294+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040440 con 0x7f63e0018390 2021-03-09T10:28:00.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.294+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:00.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.294+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:00.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.294+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:00.303 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:00.304 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:00.304 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:00.304 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:00.304 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:00.304 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:00.305 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:00.305 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:00.305 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:00.305 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:00.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.298+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055540 con 0x7f63e0018390 2021-03-09T10:28:00.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.298+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:00.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.298+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:28:00.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.298+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63e0018390 2021-03-09T10:28:00.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.497+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:00.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:00.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:00.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:00.601+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:01.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:01.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:01.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:01.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:02.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:02.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:02.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:02.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:03.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:03.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:03.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:03.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:04.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:04.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:04.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:04.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:05.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:05.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:05.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16448 cs=0 l=1 rev1=1 rx=0x7f63f007af40 tx=0x7f63f0027ee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:05.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16448 cs=0 l=1 rev1=1 rx=0x7f63f007af40 tx=0x7f63f0027ee0).stop 2021-03-09T10:28:05.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:05.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:05.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:05.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:05.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:28:05.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:05.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:05.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:05.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:05.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:05.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:05.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:05.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00133f0 con 0x7f63d8003400 2021-03-09T10:28:05.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:05.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:28:05.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:05.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066370 con 0x7f63d8003840 2021-03-09T10:28:05.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:05.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:05.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:05.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:05.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.293+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:05.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.294+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:05.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.294+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:28:05.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.294+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:05.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.294+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:05.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:05.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:05.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:05.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:05.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:05.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:05.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024c30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:05.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:05.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:28:05.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:28:05.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:05.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:05.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027c30 con 0x7f63d8003840 2021-03-09T10:28:05.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:05.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:05.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079bf0 con 0x7f63d8003840 2021-03-09T10:28:05.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:28:05.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.295+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.296+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16465 cs=0 l=1 rev1=1 rx=0x7f63fc0099d0 tx=0x7f63fc03fea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.296+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:28:05.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.296+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:05.309 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:05.301+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45645 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:05.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63d8003840 2021-03-09T10:28:05.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.301+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:05.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.301+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:05.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.301+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:05.312 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:05.312 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:05.312 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:05.312 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:05.313 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:05.313 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:05.313 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:05.313 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:05.314 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:05.314 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:05.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63d8003840 2021-03-09T10:28:05.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.305+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:05.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.305+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:28:05.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8003840 2021-03-09T10:28:05.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.498+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:05.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.601+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:05.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:05.601+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:06.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:06.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:06.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:06.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:07.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:07.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:07.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:07.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:08.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:08.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:08.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:08.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:09.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:09.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:09.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:09.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:10.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:10.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:10.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16465 cs=0 l=1 rev1=1 rx=0x7f63fc0099d0 tx=0x7f63fc03fea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:10.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=16465 cs=0 l=1 rev1=1 rx=0x7f63fc0099d0 tx=0x7f63fc03fea0).stop 2021-03-09T10:28:10.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:10.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=16465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:28:10.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:10.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:10.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:10.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:10.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:10.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:10.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:10.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:28:10.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:10.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00138d0 con 0x7f63d8009130 2021-03-09T10:28:10.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:10.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00158b0 con 0x7f63e0018390 2021-03-09T10:28:10.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:10.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:10.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:10.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:10.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:10.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:10.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:28:10.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:10.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:10.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:10.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:10.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:10.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:10.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:10.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:10.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:10.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:10.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:28:10.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:28:10.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:10.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:10.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a1a0 con 0x7f63e0018390 2021-03-09T10:28:10.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:10.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:10.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027c30 con 0x7f63e0018390 2021-03-09T10:28:10.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:28:10.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.302+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:10.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.303+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16475 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc066e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:10.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.303+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:28:10.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.303+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:10.315 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:10.307+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45686 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:10.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63e0018390 2021-03-09T10:28:10.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.308+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:10.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.308+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:10.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.308+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:10.317 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:10.317 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:10.318 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:10.319 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:10.319 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:10.319 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:10.319 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:10.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0473d0 con 0x7f63e0018390 2021-03-09T10:28:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.308+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.308+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:28:10.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc052b60 con 0x7f63e0018390 2021-03-09T10:28:10.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.499+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:10.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:10.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:10.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:10.602+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:11.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:11.602+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:11.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:11.602+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:12.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:12.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:12.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:12.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:13.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:13.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:13.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:13.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:14.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:14.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:14.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:14.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:15.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:15.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:15.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16475 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc066e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:15.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16475 cs=0 l=1 rev1=1 rx=0x7f63fc01e030 tx=0x7f63fc066e90).stop 2021-03-09T10:28:15.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:15.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:15.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:15.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:15.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:28:15.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:15.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:15.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:15.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:15.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:15.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:15.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:15.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ca30 con 0x7f63d8003400 2021-03-09T10:28:15.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:15.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fdb0 con 0x7f63d8009130 2021-03-09T10:28:15.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:15.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00133f0 con 0x7f63d8003840 2021-03-09T10:28:15.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:15.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:15.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:15.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:15.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.308+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:15.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.309+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:15.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.309+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:28:15.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.309+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:15.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.309+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:15.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.310+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:15.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.310+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:15.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.310+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:15.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:15.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:15.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:15.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:15.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:15.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:28:15.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:28:15.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:15.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:15.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029de0 con 0x7f63d8003840 2021-03-09T10:28:15.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:15.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:15.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a1a0 con 0x7f63d8003840 2021-03-09T10:28:15.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:28:15.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.311+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:15.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.312+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:28:15.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.312+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:15.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.312+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c3f0 secure :-1 s=READY pgs=16498 cs=0 l=1 rev1=1 rx=0x7f63fc067050 tx=0x7f63fc01f580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:15.323 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:15.314+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45723 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:15.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052b60 con 0x7f63d8003840 2021-03-09T10:28:15.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.314+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:15.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.314+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:15.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.314+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:15.325 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:15.325 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:15.326 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:15.326 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:15.326 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:15.326 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:15.326 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:15.327 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:15.327 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:15.327 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:15.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc012dd0 con 0x7f63d8003840 2021-03-09T10:28:15.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.316+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:15.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.316+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:28:15.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.319+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc052b60 con 0x7f63d8003840 2021-03-09T10:28:15.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.500+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:15.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:15.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:15.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:16.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:16.603+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:16.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:16.603+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:17.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:17.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:17.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:17.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:18.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:18.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:18.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:18.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:19.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:19.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:19.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:19.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:20.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:20.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:20.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c3f0 secure :-1 s=READY pgs=16498 cs=0 l=1 rev1=1 rx=0x7f63fc067050 tx=0x7f63fc01f580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:20.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c3f0 secure :-1 s=READY pgs=16498 cs=0 l=1 rev1=1 rx=0x7f63fc067050 tx=0x7f63fc01f580).stop 2021-03-09T10:28:20.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:20.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:20.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c3f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:20.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c3f0 unknown :-1 s=CLOSED pgs=16498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:20.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:28:20.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:20.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:20.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:20.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:20.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:20.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:20.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:20.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023ac0 con 0x7f63d8003400 2021-03-09T10:28:20.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:20.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:28:20.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:20.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00237d0 con 0x7f63e0018390 2021-03-09T10:28:20.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:20.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:20.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:20.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:20.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.316+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:20.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:20.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.317+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:28:20.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.317+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:20.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.317+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:20.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:20.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:20.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:20.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:20.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:20.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:20.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:20.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:20.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:28:20.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:28:20.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:20.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:20.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T10:28:20.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:20.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:20.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00162e0 con 0x7f63e0018390 2021-03-09T10:28:20.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:28:20.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:20.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16511 cs=0 l=1 rev1=1 rx=0x7f63f004c370 tx=0x7f63f0003500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:20.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:28:20.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.318+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:20.332 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:20.321+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45764 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:20.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0018390 2021-03-09T10:28:20.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.321+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:20.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.321+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:20.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.321+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:20.333 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:20.334 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:20.334 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:20.334 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:20.334 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:20.334 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:20.335 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:20.335 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:20.335 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:20.335 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:20.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T10:28:20.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.323+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:20.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.323+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:28:20.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00603b0 con 0x7f63e0018390 2021-03-09T10:28:20.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.501+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:20.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:20.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:20.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:20.604+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:21.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:21.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:21.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:21.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:22.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:22.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:22.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:22.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:23.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:23.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:23.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:23.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:24.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:24.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:24.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:24.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:25.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:25.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:25.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16511 cs=0 l=1 rev1=1 rx=0x7f63f004c370 tx=0x7f63f0003500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:25.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16511 cs=0 l=1 rev1=1 rx=0x7f63f004c370 tx=0x7f63f0003500).stop 2021-03-09T10:28:25.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:25.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:25.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:25.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:25.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:28:25.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:25.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:25.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:25.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:25.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:25.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:25.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:25.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00133f0 con 0x7f63d8003400 2021-03-09T10:28:25.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:25.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fdb0 con 0x7f63d8009130 2021-03-09T10:28:25.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:25.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00098e0 con 0x7f63d8003840 2021-03-09T10:28:25.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.321+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:25.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.322+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:25.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.322+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:25.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.322+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:25.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.322+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:25.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.322+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:25.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.322+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:28:25.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.322+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:25.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.322+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:25.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.323+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:25.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.323+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:25.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.323+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:25.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:25.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:25.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:25.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:25.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:25.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:28:25.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:28:25.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:25.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:25.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029c50 con 0x7f63d8003840 2021-03-09T10:28:25.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:25.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:25.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63d8003840 2021-03-09T10:28:25.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:28:25.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.324+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:25.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.325+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=16522 cs=0 l=1 rev1=1 rx=0x7f63f007af40 tx=0x7f63f0006680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:25.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.325+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:28:25.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.325+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:25.339 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:25.327+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45802 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:25.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007be90 con 0x7f63d8003840 2021-03-09T10:28:25.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.328+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:25.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.328+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:25.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.328+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:25.341 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:25.342 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:25.342 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:25.342 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:25.343 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:25.343 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:25.343 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:25.343 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:25.343 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:25.344 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:25.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007be90 con 0x7f63d8003840 2021-03-09T10:28:25.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.331+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:25.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.331+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:28:25.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.331+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8003840 2021-03-09T10:28:25.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.502+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:25.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:25.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:25.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:26.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:26.604+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:26.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:26.604+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:27.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:27.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:27.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:27.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:28.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:28.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:28.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:28.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:29.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:29.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:29.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:29.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:30.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:28:30.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:30.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=16522 cs=0 l=1 rev1=1 rx=0x7f63f007af40 tx=0x7f63f0006680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:30.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=16522 cs=0 l=1 rev1=1 rx=0x7f63f007af40 tx=0x7f63f0006680).stop 2021-03-09T10:28:30.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:30.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:30.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:30.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=CLOSED pgs=16522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:30.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:28:30.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:30.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:30.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:30.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:30.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:30.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:30.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:30.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:28:30.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:30.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:28:30.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:30.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068ea0 con 0x7f63e0018390 2021-03-09T10:28:30.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:30.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:30.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.328+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:30.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:30.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:30.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:30.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.329+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:28:30.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.329+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:30.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.329+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:30.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.330+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:30.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.330+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.330+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:30.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:30.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:30.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:30.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:28:30.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:28:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:28:30.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:30.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:30.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029c50 con 0x7f63e0018390 2021-03-09T10:28:30.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:28:30.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:30.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16531 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:30.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:28:30.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.331+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:30.345 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:30.334+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45842 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:30.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:28:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.335+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.335+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.335+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:30.346 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:30.347 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:30.347 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:30.347 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:30.347 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:30.348 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:30.348 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:30.348 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:30.348 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:30.348 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:30.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.337+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:28:30.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.337+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:30.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.337+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:28:30.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63e0018390 2021-03-09T10:28:30.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.503+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:30.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:30.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:30.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:30.605+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:31.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:31.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:31.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:31.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:32.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:32.605+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:32.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:32.605+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:33.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:33.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:33.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:33.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:34.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:34.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:34.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:34.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:35.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:35.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:35.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16531 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:35.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16531 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004db20).stop 2021-03-09T10:28:35.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:35.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:35.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:35.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:35.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:28:35.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:35.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:35.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:35.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:35.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:35.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:35.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:35.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:28:35.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:35.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065b20 con 0x7f63d8009130 2021-03-09T10:28:35.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:35.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021630 con 0x7f63d8003840 2021-03-09T10:28:35.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.334+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:35.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.335+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:35.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.335+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:35.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.335+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:35.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.335+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:35.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.335+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:35.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:28:35.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:35.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:35.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:35.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:35.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:35.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:35.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:35.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:35.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:35.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:35.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:28:35.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:28:35.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:35.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:35.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010cf0 con 0x7f63d8003840 2021-03-09T10:28:35.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:35.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:35.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029de0 con 0x7f63d8003840 2021-03-09T10:28:35.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:28:35.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.336+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:35.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.337+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=16543 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc01f340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:35.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.337+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:28:35.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.337+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:35.350 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:35.341+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45880 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:35.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.342+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:28:35.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.342+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:35.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.342+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:35.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.342+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:35.352 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:35.352 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:35.352 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:35.352 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:35.353 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:35.353 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:35.353 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:35.353 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:35.353 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:35.354 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:35.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63d8003840 2021-03-09T10:28:35.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.348+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:35.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8003840 2021-03-09T10:28:35.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.348+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:28:35.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.504+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:35.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:35.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:35.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:36.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:36.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:36.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:36.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:37.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:37.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:37.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:37.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:38.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:38.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:38.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:38.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:39.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:39.606+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:39.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:39.606+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:40.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:40.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:40.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=16543 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc01f340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:40.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=16543 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc01f340).stop 2021-03-09T10:28:40.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:40.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:40.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:40.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=CLOSED pgs=16543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:40.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:28:40.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:40.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:40.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:40.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:40.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:40.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:40.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:40.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022f40 con 0x7f63d8003400 2021-03-09T10:28:40.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:40.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:28:40.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:40.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:28:40.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.341+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:40.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:40.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:40.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:40.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:40.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:40.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:28:40.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:40.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:40.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:40.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:40.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.342+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:40.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:40.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:40.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:40.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:40.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:40.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:28:40.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:28:40.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:40.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:40.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a1a0 con 0x7f63e0018390 2021-03-09T10:28:40.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:40.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:40.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:28:40.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:28:40.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.343+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:40.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.344+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:28:40.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.344+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16557 cs=0 l=1 rev1=1 rx=0x7f63f0048590 tx=0x7f63f007cf60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:40.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.344+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:40.356 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:40.347+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45920 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:40.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63e0018390 2021-03-09T10:28:40.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.348+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:40.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.348+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:40.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.348+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:40.358 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:40.358 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:40.358 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:40.358 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:40.359 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:40.359 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:40.359 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:40.359 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:40.359 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:40.359 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:40.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63e0018390 2021-03-09T10:28:40.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.348+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:40.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.348+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:28:40.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63e0018390 2021-03-09T10:28:40.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.505+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:40.607+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:41.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:41.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:41.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:42.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:42.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:42.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:42.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:43.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:43.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:43.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:43.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:44.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:44.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:45.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:44.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:45.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:45.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:45.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16557 cs=0 l=1 rev1=1 rx=0x7f63f0048590 tx=0x7f63f007cf60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:45.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16557 cs=0 l=1 rev1=1 rx=0x7f63f0048590 tx=0x7f63f007cf60).stop 2021-03-09T10:28:45.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:45.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:45.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:45.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:45.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:28:45.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:45.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:45.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:45.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:45.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:45.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:45.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:45.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8003400 2021-03-09T10:28:45.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:45.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:28:45.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:45.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003840 2021-03-09T10:28:45.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:45.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:45.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:45.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:45.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.348+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:45.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:45.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.349+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:28:45.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.349+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:45.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.349+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:45.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:45.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:45.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:45.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:45.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:45.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:45.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:45.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:45.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:28:45.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:28:45.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:45.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:45.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f63d8003840 2021-03-09T10:28:45.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:45.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:45.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a1a0 con 0x7f63d8003840 2021-03-09T10:28:45.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:28:45.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.350+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:45.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067130 secure :-1 s=READY pgs=16574 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f0027ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:45.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.351+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:28:45.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.351+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:45.707 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:45.354+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45958 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:45.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b480 con 0x7f63d8003840 2021-03-09T10:28:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.355+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.355+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.355+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:45.711 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.357+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b480 con 0x7f63d8003840 2021-03-09T10:28:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.357+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.357+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:28:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:28:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.506+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:45.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:45.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:46.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:46.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:46.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:46.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:47.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:47.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:47.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:47.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:48.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:48.607+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:48.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:48.607+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:49.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:49.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:49.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:49.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:50.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:50.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:50.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067130 secure :-1 s=READY pgs=16574 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f0027ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:50.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067130 secure :-1 s=READY pgs=16574 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f0027ea0).stop 2021-03-09T10:28:50.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:50.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:50.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:50.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067130 unknown :-1 s=CLOSED pgs=16574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:50.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:28:50.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:50.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:50.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:50.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:50.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:50.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:50.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:50.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:28:50.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:50.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8009130 2021-03-09T10:28:50.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:50.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63e0018390 2021-03-09T10:28:50.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:50.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:50.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:50.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:50.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.355+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:50.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:50.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.356+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:28:50.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.356+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:50.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.356+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:50.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.357+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:50.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.357+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:50.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.357+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:50.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:50.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:50.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:50.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:50.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:50.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:28:50.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:28:50.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:50.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:50.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d470 con 0x7f63e0018390 2021-03-09T10:28:50.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:50.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:50.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T10:28:50.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:28:50.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:50.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16591 cs=0 l=1 rev1=1 rx=0x7f63f007f580 tx=0x7f63f0023520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:50.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:28:50.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.358+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:50.370 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:50.362+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 45998 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:50.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.363+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028730 con 0x7f63e0018390 2021-03-09T10:28:50.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.363+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:50.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.363+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:50.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.363+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:50.372 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:50.373 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:50.373 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:50.373 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:50.373 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:50.373 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:50.374 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:50.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028730 con 0x7f63e0018390 2021-03-09T10:28:50.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.365+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:50.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.365+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:28:50.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.366+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63e0018390 2021-03-09T10:28:50.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.507+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:50.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:50.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:50.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:50.608+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:51.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:51.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:51.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:51.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:52.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:52.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:52.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:52.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:53.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:53.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:53.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:53.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:54.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:54.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:54.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:54.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:55.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:28:55.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:28:55.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16591 cs=0 l=1 rev1=1 rx=0x7f63f007f580 tx=0x7f63f0023520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:28:55.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16591 cs=0 l=1 rev1=1 rx=0x7f63f007f580 tx=0x7f63f0023520).stop 2021-03-09T10:28:55.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:55.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:28:55.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:28:55.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:28:55.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:28:55.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:55.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:28:55.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:55.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:28:55.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:28:55.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:28:55.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:55.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:28:55.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:55.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bdc0 con 0x7f63d8009130 2021-03-09T10:28:55.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:28:55.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00147e0 con 0x7f63d8003840 2021-03-09T10:28:55.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:28:55.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:28:55.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:55.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:28:55.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:28:55.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:28:55.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:28:55.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:28:55.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.363+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:28:55.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.364+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:28:55.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.364+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:28:55.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.364+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:28:55.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:28:55.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:55.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:55.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:28:55.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:28:55.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:28:55.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:28:55.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:28:55.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:55.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006bf00 con 0x7f63d8003840 2021-03-09T10:28:55.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:28:55.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:28:55.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010cf0 con 0x7f63d8003840 2021-03-09T10:28:55.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:28:55.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:55.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:28:55.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:28:55.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.365+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017470 secure :-1 s=READY pgs=16603 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc03fea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:28:55.378 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:28:55.370+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46037 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:28:55.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8003840 2021-03-09T10:28:55.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.371+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:28:55.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.371+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:28:55.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.371+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:28:55.380 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:28:55.380 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:28:55.380 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:55.380 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:28:55.381 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:28:55.381 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:28:55.381 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:28:55.381 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:28:55.382 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:28:55.382 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:28:55.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8003840 2021-03-09T10:28:55.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.375+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:28:55.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.375+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:28:55.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.376+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8003840 2021-03-09T10:28:55.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.508+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:28:55.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:55.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:55.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:56.608+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:56.608+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:57.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:57.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:57.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:57.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:58.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:58.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:58.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:58.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:28:59.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:59.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:28:59.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:28:59.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:00.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:00.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:00.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017470 secure :-1 s=READY pgs=16603 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc03fea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:00.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017470 secure :-1 s=READY pgs=16603 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc03fea0).stop 2021-03-09T10:29:00.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:00.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:00.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:00.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017470 unknown :-1 s=CLOSED pgs=16603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:00.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:29:00.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:00.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:00.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:00.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8003400 2021-03-09T10:29:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:29:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:00.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:29:00.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:00.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:00.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:00.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:00.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:00.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:29:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.371+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:00.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.372+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.372+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.372+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:00.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:00.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:00.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:00.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:00.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:29:00.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:29:00.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:00.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:00.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b500 con 0x7f63e0018390 2021-03-09T10:29:00.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:00.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:00.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006bf00 con 0x7f63e0018390 2021-03-09T10:29:00.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:29:00.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:00.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16615 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc04bb10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:00.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:29:00.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.373+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:00.386 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:00.376+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46077 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:00.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f63e0018390 2021-03-09T10:29:00.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.377+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:00.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.377+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:00.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.377+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:00.388 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:00.388 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:00.389 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:00.389 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:00.389 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:00.389 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:00.390 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:00.390 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:00.390 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:00.390 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:00.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f63e0018390 2021-03-09T10:29:00.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.377+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:00.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.377+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:29:00.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc009520 con 0x7f63e0018390 2021-03-09T10:29:00.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.509+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:00.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:00.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:00.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:00.609+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:01.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:01.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:01.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:01.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:02.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:02.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:02.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:02.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:03.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:03.609+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:03.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:03.609+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:04.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:04.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:04.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:04.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:05.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:05.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:05.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16615 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc04bb10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:05.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16615 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc04bb10).stop 2021-03-09T10:29:05.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:05.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:05.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:05.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:05.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:29:05.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:05.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:05.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:05.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:05.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:05.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:05.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:05.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068470 con 0x7f63d8003400 2021-03-09T10:29:05.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:05.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b530 con 0x7f63d8009130 2021-03-09T10:29:05.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:05.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003840 2021-03-09T10:29:05.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:05.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:05.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:05.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.377+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:05.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:05.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:05.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:29:05.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:05.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:05.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:05.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:05.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:05.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:05.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:05.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:05.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:05.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:05.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:29:05.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:29:05.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:05.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:05.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00162e0 con 0x7f63d8003840 2021-03-09T10:29:05.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:05.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:05.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d470 con 0x7f63d8003840 2021-03-09T10:29:05.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:29:05.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.378+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:05.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.379+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=16624 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0003500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:05.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.379+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:29:05.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.379+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:05.397 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:05.384+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46116 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:05.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.386+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8003840 2021-03-09T10:29:05.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.386+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:05.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.386+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:05.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.386+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:05.399 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:05.399 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:05.399 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:05.399 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:05.400 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:05.400 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:05.400 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:05.400 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:05.400 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:05.401 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:05.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8003840 2021-03-09T10:29:05.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.387+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:05.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.387+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:29:05.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:29:05.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.510+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:05.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:05.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:05.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:06.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:06.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:06.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:06.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:07.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:07.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:07.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:07.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:08.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:08.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:08.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:09.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:09.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:09.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:10.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=16624 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0003500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=16624 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0003500).stop 2021-03-09T10:29:10.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:10.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:10.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:10.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=CLOSED pgs=16624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:10.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:29:10.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:10.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:10.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:10.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:10.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:10.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:10.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:10.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002edd0 con 0x7f63d8003400 2021-03-09T10:29:10.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:10.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:29:10.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:10.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63e0018390 2021-03-09T10:29:10.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:10.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:10.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:10.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:10.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:10.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:10.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:29:10.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:10.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:10.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.385+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:10.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:10.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:10.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:10.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:10.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:10.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:10.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:10.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:29:10.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:29:10.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:10.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:10.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T10:29:10.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:10.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:10.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b500 con 0x7f63e0018390 2021-03-09T10:29:10.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:29:10.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.386+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:10.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.387+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:29:10.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.387+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16641 cs=0 l=1 rev1=1 rx=0x7f63fc067050 tx=0x7f63fc00bf70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:10.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.387+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:10.402 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:10.392+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46155 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:10.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.394+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c630 con 0x7f63e0018390 2021-03-09T10:29:10.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.394+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:10.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.394+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:10.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.394+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:10.403 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:10.404 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:10.404 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:10.404 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:10.404 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:10.404 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:10.405 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:10.405 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:10.405 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:10.406 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:10.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02c400 con 0x7f63e0018390 2021-03-09T10:29:10.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.397+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:10.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.397+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:29:10.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.398+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63e0018390 2021-03-09T10:29:10.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.511+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:10.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:10.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:10.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:10.611+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:11.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:11.610+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:11.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:11.610+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:12.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:12.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:12.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:13.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:13.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:13.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:13.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:14.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:14.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:14.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:14.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:15.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:15.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:15.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16641 cs=0 l=1 rev1=1 rx=0x7f63fc067050 tx=0x7f63fc00bf70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:15.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16641 cs=0 l=1 rev1=1 rx=0x7f63fc067050 tx=0x7f63fc00bf70).stop 2021-03-09T10:29:15.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:15.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:15.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:15.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:15.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:29:15.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:15.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:15.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:15.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:15.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:15.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:15.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:15.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:29:15.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:15.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:29:15.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:15.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:29:15.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:15.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:15.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:15.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:15.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:15.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:15.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:29:15.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:15.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:15.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:15.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:15.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.393+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:15.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:15.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:15.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:15.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:15.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:15.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:29:15.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:29:15.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:15.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:15.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012120 con 0x7f63d8003840 2021-03-09T10:29:15.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:15.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:15.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00162e0 con 0x7f63d8003840 2021-03-09T10:29:15.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:29:15.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:15.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:29:15.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018770 secure :-1 s=READY pgs=16652 cs=0 l=1 rev1=1 rx=0x7f63f007f5b0 tx=0x7f63f007f720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:15.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.394+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:15.408 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:15.399+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46195 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:15.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066f00 con 0x7f63d8003840 2021-03-09T10:29:15.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.399+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:15.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.399+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:15.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.399+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:15.410 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:15.410 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:15.411 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:15.411 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:15.411 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:15.411 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:15.412 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:15.412 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:15.412 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:15.412 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.399+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:29:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.399+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.399+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:29:15.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:29:15.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.512+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:15.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:15.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:15.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:16.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:16.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:16.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:16.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:17.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:17.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:17.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:17.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:18.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:18.611+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:18.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:18.611+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:19.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:19.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:19.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:19.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:20.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0018770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:20.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0018770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:20.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018770 secure :-1 s=READY pgs=16652 cs=0 l=1 rev1=1 rx=0x7f63f007f5b0 tx=0x7f63f007f720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:20.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018770 secure :-1 s=READY pgs=16652 cs=0 l=1 rev1=1 rx=0x7f63f007f5b0 tx=0x7f63f007f720).stop 2021-03-09T10:29:20.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:20.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:20.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0018770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:20.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018770 unknown :-1 s=CLOSED pgs=16652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:20.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:29:20.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:20.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:20.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:20.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:20.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:20.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:20.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:20.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013290 con 0x7f63d8003400 2021-03-09T10:29:20.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:20.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:29:20.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:20.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00106f0 con 0x7f63e0018390 2021-03-09T10:29:20.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:20.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:20.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:20.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:20.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.400+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:20.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.401+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:20.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.401+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:29:20.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.401+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:20.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.401+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:20.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:20.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:20.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:20.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:20.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:20.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:20.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:20.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:20.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:29:20.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:29:20.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:20.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:20.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0000ad0 con 0x7f63e0018390 2021-03-09T10:29:20.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:20.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:20.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012120 con 0x7f63e0018390 2021-03-09T10:29:20.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:29:20.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:20.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:29:20.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16674 cs=0 l=1 rev1=1 rx=0x7f63f007ef20 tx=0x7f63f00066f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:20.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.403+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:20.415 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:20.406+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46233 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:20.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.406+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:29:20.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.406+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:20.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.406+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:20.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.406+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:20.416 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:20.417 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:20.417 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:20.417 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:20.417 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:20.418 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:20.418 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:20.418 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:20.418 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:20.418 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:20.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.407+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:29:20.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.407+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:20.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.407+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:29:20.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.408+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e0018390 2021-03-09T10:29:20.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.513+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:20.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:20.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:20.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:20.612+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:21.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:21.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:21.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:21.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:22.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:22.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:22.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:22.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:23.612+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:23.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:23.612+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:24.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:24.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:24.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:24.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:25.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.407+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:25.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.407+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:25.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.407+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16674 cs=0 l=1 rev1=1 rx=0x7f63f007ef20 tx=0x7f63f00066f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:25.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.407+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16674 cs=0 l=1 rev1=1 rx=0x7f63f007ef20 tx=0x7f63f00066f0).stop 2021-03-09T10:29:25.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:25.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:25.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:25.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:25.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:29:25.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:25.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:25.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:25.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:25.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:25.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:25.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:25.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:29:25.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:25.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:29:25.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:25.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003840 2021-03-09T10:29:25.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:25.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:25.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:25.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:25.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.408+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:25.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:25.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.409+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:29:25.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.409+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:25.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.409+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:25.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.409+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:25.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.409+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:25.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.409+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:25.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:25.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:25.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:25.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:25.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:25.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:29:25.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:29:25.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:25.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:25.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a1a0 con 0x7f63d8003840 2021-03-09T10:29:25.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:25.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:25.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0000ad0 con 0x7f63d8003840 2021-03-09T10:29:25.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:29:25.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:25.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:29:25.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:25.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.410+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 secure :-1 s=READY pgs=16688 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f00885d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:25.423 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:25.413+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46274 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:25.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.413+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b6c0 con 0x7f63d8003840 2021-03-09T10:29:25.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.414+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:25.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.414+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:25.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.414+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:25.425 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:25.425 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:25.425 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:25.425 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:25.426 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:25.426 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:25.426 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:25.426 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:25.427 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:25.427 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:25.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:29:25.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.417+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:25.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.417+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:29:25.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.418+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63d8003840 2021-03-09T10:29:25.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.514+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:25.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:25.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:25.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:26.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:26.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:26.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:26.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:27.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:27.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:27.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:27.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:28.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:28.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:28.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:28.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:29.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:29.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:29.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:29.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:30.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0018fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:30.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0018fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:30.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 secure :-1 s=READY pgs=16688 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f00885d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:30.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 secure :-1 s=READY pgs=16688 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f00885d0).stop 2021-03-09T10:29:30.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:30.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:30.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0018fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:30.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 unknown :-1 s=CLOSED pgs=16688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:30.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:29:30.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:30.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:30.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:30.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:30.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:30.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.415+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:30.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:30.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b550 con 0x7f63d8003400 2021-03-09T10:29:30.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:30.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:29:30.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:30.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:29:30.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:30.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:30.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:30.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:30.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.416+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:30.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.417+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:30.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.417+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:29:30.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.417+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:30.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.417+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:30.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.417+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:30.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.417+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:30.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.417+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:30.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:30.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:30.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:30.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:30.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:30.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:29:30.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:29:30.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:30.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:30.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028b60 con 0x7f63e0018390 2021-03-09T10:29:30.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:30.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:30.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a1a0 con 0x7f63e0018390 2021-03-09T10:29:30.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:29:30.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:30.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16703 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0027e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:30.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:29:30.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.418+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:30.433 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:30.421+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46312 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:30.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.422+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002f1d0 con 0x7f63e0018390 2021-03-09T10:29:30.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.422+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:30.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.422+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:30.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.422+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:30.435 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:30.435 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:30.436 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:30.436 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:30.436 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:30.436 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:30.436 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:30.437 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:30.437 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:30.437 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:30.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.424+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:29:30.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.424+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:30.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.424+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:29:30.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.427+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63e0018390 2021-03-09T10:29:30.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.515+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:30.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:30.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:30.613+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:31.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:31.613+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:31.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:31.613+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:32.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:32.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:32.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:32.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:33.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:33.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:33.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:33.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:34.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:34.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:34.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:34.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:35.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:35.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:35.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16703 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0027e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:35.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16703 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0027e30).stop 2021-03-09T10:29:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:35.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:35.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:29:35.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:35.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:35.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:35.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:35.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:35.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:35.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:35.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc30 con 0x7f63d8003400 2021-03-09T10:29:35.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:35.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013290 con 0x7f63d8009130 2021-03-09T10:29:35.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:35.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e2b0 con 0x7f63d8003840 2021-03-09T10:29:35.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.423+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:35.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.424+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:35.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.424+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:35.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.424+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:35.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.424+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:35.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:35.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:29:35.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:35.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:35.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:35.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:35.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:35.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:35.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:35.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:35.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:35.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:35.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:29:35.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:29:35.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:35.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:35.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010cf0 con 0x7f63d8003840 2021-03-09T10:29:35.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:35.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:35.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63d8003840 2021-03-09T10:29:35.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:29:35.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:35.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 secure :-1 s=READY pgs=16715 cs=0 l=1 rev1=1 rx=0x7f63fc008c00 tx=0x7f63fc04bb80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:35.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.426+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:29:35.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.426+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:35.438 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:35.430+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46351 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:35.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c400 con 0x7f63d8003840 2021-03-09T10:29:35.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.431+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:35.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.431+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:35.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.431+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:35.440 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:35.440 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:35.441 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:35.441 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:35.441 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:35.441 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:35.442 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:35.442 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:35.442 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:35.442 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:35.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc008a40 con 0x7f63d8003840 2021-03-09T10:29:35.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.434+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:35.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.434+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:29:35.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.435+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:29:35.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.516+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:35.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:35.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:35.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:36.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:36.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:36.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:36.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:37.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:37.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:37.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:37.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:38.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:38.614+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:38.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:38.614+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:39.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:39.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:39.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:39.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:40.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.430+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0018fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:40.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.430+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0018fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:40.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.430+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 secure :-1 s=READY pgs=16715 cs=0 l=1 rev1=1 rx=0x7f63fc008c00 tx=0x7f63fc04bb80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:40.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.430+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 secure :-1 s=READY pgs=16715 cs=0 l=1 rev1=1 rx=0x7f63fc008c00 tx=0x7f63fc04bb80).stop 2021-03-09T10:29:40.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:40.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:40.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0018fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:40.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0018fa0 unknown :-1 s=CLOSED pgs=16715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:40.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:29:40.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:40.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:40.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:40.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:40.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:40.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:40.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:40.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:29:40.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:40.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:29:40.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:40.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63e0018390 2021-03-09T10:29:40.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:40.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:40.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:40.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:40.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:40.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:40.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:29:40.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:40.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.431+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:40.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.432+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:40.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.432+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:40.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.432+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:40.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:40.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:40.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:40.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:40.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:40.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:29:40.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:29:40.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:40.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:40.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T10:29:40.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:40.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:40.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028b60 con 0x7f63e0018390 2021-03-09T10:29:40.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:29:40.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:40.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:29:40.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:40.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.433+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16727 cs=0 l=1 rev1=1 rx=0x7f63f0055630 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:40.445 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:40.437+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46391 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:40.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.439+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036d80 con 0x7f63e0018390 2021-03-09T10:29:40.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.439+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:40.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.439+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:40.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.439+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:40.447 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:40.447 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:40.447 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:40.448 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:40.448 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:40.448 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:40.448 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:40.448 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:40.449 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:40.449 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:40.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.439+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0036d80 con 0x7f63e0018390 2021-03-09T10:29:40.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.439+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:40.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.439+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:29:40.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.440+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63e0018390 2021-03-09T10:29:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.517+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:40.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:40.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:40.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:40.615+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:41.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:41.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:41.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:41.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:42.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:42.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:42.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:42.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:43.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:43.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:43.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:43.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:44.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:44.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:44.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:44.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:45.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:29:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16727 cs=0 l=1 rev1=1 rx=0x7f63f0055630 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16727 cs=0 l=1 rev1=1 rx=0x7f63f0055630 tx=0x7f63f004db20).stop 2021-03-09T10:29:45.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:29:45.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:45.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019c20 con 0x7f63d8003400 2021-03-09T10:29:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:45.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e2b0 con 0x7f63d8009130 2021-03-09T10:29:45.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:45.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:29:45.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:45.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:45.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:45.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:45.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.438+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:45.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.439+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:45.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.439+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:29:45.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.439+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:45.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.439+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:45.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.439+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:45.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.439+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:45.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.439+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:45.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:45.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:45.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:45.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:45.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:45.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:29:45.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:29:45.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:45.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:45.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028b60 con 0x7f63d8003840 2021-03-09T10:29:45.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:45.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:45.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63d8003840 2021-03-09T10:29:45.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:29:45.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.440+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:45.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.441+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d260 secure :-1 s=READY pgs=16739 cs=0 l=1 rev1=1 rx=0x7f63f0028810 tx=0x7f63f0041f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:45.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.442+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:29:45.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.442+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:45.452 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:45.445+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46432 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:45.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00284b0 con 0x7f63d8003840 2021-03-09T10:29:45.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.450+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:45.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.450+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:45.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.450+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:45.454 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:45.454 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:45.454 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:45.455 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:45.455 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:45.455 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:45.455 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:45.455 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:45.456 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:45.456 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:45.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066f00 con 0x7f63d8003840 2021-03-09T10:29:45.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.450+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:45.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.450+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:29:45.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63d8003840 2021-03-09T10:29:45.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.518+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:45.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.615+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:45.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:45.615+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:46.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:46.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:46.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:46.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:47.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:47.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:47.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:47.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:48.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:48.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:48.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:48.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:49.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:49.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:49.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:49.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:50.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.445+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:50.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.445+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:50.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.445+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d260 secure :-1 s=READY pgs=16739 cs=0 l=1 rev1=1 rx=0x7f63f0028810 tx=0x7f63f0041f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:50.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d260 secure :-1 s=READY pgs=16739 cs=0 l=1 rev1=1 rx=0x7f63f0028810 tx=0x7f63f0041f00).stop 2021-03-09T10:29:50.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:50.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:50.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:50.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d260 unknown :-1 s=CLOSED pgs=16739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:50.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:29:50.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:50.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:50.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:50.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:50.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:50.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:50.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:50.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019a20 con 0x7f63d8003400 2021-03-09T10:29:50.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:50.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8009130 2021-03-09T10:29:50.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:50.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63e0018390 2021-03-09T10:29:50.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:50.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:50.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:50.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:50.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:50.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:29:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.446+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:50.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:50.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:50.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:50.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:29:50.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:29:50.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:50.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:50.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.447+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00266d0 con 0x7f63e0018390 2021-03-09T10:29:50.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.448+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:50.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.448+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.448+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028b60 con 0x7f63e0018390 2021-03-09T10:29:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.448+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:29:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.448+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:50.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.448+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16749 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0066f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.448+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:29:50.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.448+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:50.460 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:50.452+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46470 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:50.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63e0018390 2021-03-09T10:29:50.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.452+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:50.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.452+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.452+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:50.462 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:50.463 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:50.463 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:50.463 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:50.463 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:50.464 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:50.464 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:50.464 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:50.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.453+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:29:50.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.453+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:50.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.453+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:29:50.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.454+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63e0018390 2021-03-09T10:29:50.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.519+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:50.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:50.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:50.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:50.616+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:51.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:51.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:51.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:51.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:52.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:52.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:52.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:52.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:53.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:53.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:53.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:53.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:54.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:54.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:54.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:54.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:55.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:29:55.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:29:55.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16749 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0066f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:29:55.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16749 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0066f00).stop 2021-03-09T10:29:55.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:55.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:29:55.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:29:55.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:29:55.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:29:55.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:55.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:29:55.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:55.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:29:55.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:29:55.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:29:55.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:55.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:29:55.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:55.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:29:55.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:29:55.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019a20 con 0x7f63d8003840 2021-03-09T10:29:55.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:29:55.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:29:55.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:55.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:29:55.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.452+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:29:55.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.453+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:29:55.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.453+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:29:55.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.453+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:29:55.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.453+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:29:55.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.453+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:29:55.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.453+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:29:55.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.453+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:29:55.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:29:55.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:55.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:55.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:29:55.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:29:55.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:29:55.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:29:55.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:29:55.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:55.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0028240 con 0x7f63d8003840 2021-03-09T10:29:55.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:29:55.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:29:55.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00266d0 con 0x7f63d8003840 2021-03-09T10:29:55.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:29:55.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.454+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:55.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.455+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=16768 cs=0 l=1 rev1=1 rx=0x7f63f0067040 tx=0x7f63f0006680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:29:55.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.455+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:29:55.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.455+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:29:55.469 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:29:55.458+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46511 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:29:55.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0031500 con 0x7f63d8003840 2021-03-09T10:29:55.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.459+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:29:55.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.459+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:29:55.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.459+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:29:55.471 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:29:55.471 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:29:55.471 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:55.471 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:29:55.472 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:29:55.472 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:29:55.472 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:29:55.472 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:29:55.473 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:29:55.473 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:29:55.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a660 con 0x7f63d8003840 2021-03-09T10:29:55.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.461+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:29:55.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.461+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:29:55.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.464+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00312f0 con 0x7f63d8003840 2021-03-09T10:29:55.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.519+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:29:55.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:55.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:55.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:56.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:56.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:56.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:56.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:57.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:57.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:57.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:57.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:58.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:58.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:58.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:58.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:29:59.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:59.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:29:59.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:29:59.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:00.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:00.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:00.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=16768 cs=0 l=1 rev1=1 rx=0x7f63f0067040 tx=0x7f63f0006680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:00.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=16768 cs=0 l=1 rev1=1 rx=0x7f63f0067040 tx=0x7f63f0006680).stop 2021-03-09T10:30:00.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:00.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:00.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:00.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=CLOSED pgs=16768 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:00.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:30:00.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:00.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:00.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:00.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:00.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:00.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:00.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:00.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8003400 2021-03-09T10:30:00.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:00.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00696b0 con 0x7f63d8009130 2021-03-09T10:30:00.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:00.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c400 con 0x7f63e0018390 2021-03-09T10:30:00.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:00.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:00.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:00.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:00.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.459+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:00.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.460+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:00.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.460+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:30:00.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.460+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:00.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.460+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:00.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.460+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:00.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.460+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:00.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.460+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:00.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:00.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:00.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:00.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:00.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:00.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:00.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:00.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:00.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:00.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a1a0 con 0x7f63e0018390 2021-03-09T10:30:00.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:00.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:00.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0028240 con 0x7f63e0018390 2021-03-09T10:30:00.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:00.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:00.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16778 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f0036dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:00.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:30:00.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.461+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:00.474 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:00.465+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46549 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:00.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63e0018390 2021-03-09T10:30:00.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.467+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:00.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.467+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:00.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.467+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:00.475 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:00.476 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:00.476 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:00.476 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:00.476 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:00.476 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:00.477 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:00.477 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:00.477 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:00.477 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:00.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63e0018390 2021-03-09T10:30:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.468+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.468+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:30:00.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.471+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042320 con 0x7f63e0018390 2021-03-09T10:30:00.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.520+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:00.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.616+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:00.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.616+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:00.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:00.616+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:01.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:01.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:01.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:01.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:02.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:02.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:02.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:02.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:03.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:03.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:03.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:03.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:04.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:04.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:04.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:05.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.465+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:05.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.465+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:05.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.465+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16778 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f0036dc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:05.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16778 cs=0 l=1 rev1=1 rx=0x7f63f003ee80 tx=0x7f63f0036dc0).stop 2021-03-09T10:30:05.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:05.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:05.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:05.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:05.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:30:05.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:05.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:05.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:05.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:05.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:05.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:05.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:05.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00166d0 con 0x7f63d8003400 2021-03-09T10:30:05.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:05.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000bf30 con 0x7f63d8009130 2021-03-09T10:30:05.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:05.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:30:05.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:05.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:05.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:05.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:05.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:05.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:05.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:30:05.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:05.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.466+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:05.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:05.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:05.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:05.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:05.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:05.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:05.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:05.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:05.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:05.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:05.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:05.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:05.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068e60 con 0x7f63d8003840 2021-03-09T10:30:05.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:05.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:05.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a1a0 con 0x7f63d8003840 2021-03-09T10:30:05.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:05.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.467+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:05.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.468+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:30:05.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.468+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:05.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.468+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=16789 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f005f2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:05.482 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:05.473+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46590 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:05.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:30:05.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.473+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:05.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.473+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:05.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.473+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:05.485 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:05.485 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:05.486 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:05.486 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:05.487 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:05.487 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:05.488 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:05.488 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:05.488 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:05.489 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:05.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:30:05.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.476+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:05.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.476+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:30:05.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.477+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0012350 con 0x7f63d8003840 2021-03-09T10:30:05.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.521+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:05.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:05.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:05.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:06.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:06.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:06.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:06.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:07.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:07.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:07.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:07.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:08.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:08.617+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:08.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:08.617+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:09.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:09.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:09.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:09.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:10.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:10.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:10.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=16789 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f005f2e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:10.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=16789 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f005f2e0).stop 2021-03-09T10:30:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:10.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=CLOSED pgs=16789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:10.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:30:10.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:10.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:10.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:10.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:10.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00248e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:10.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:10.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:10.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002cc60 con 0x7f63d8003400 2021-03-09T10:30:10.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:10.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028860 con 0x7f63d8009130 2021-03-09T10:30:10.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:10.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d6f0 con 0x7f63e0018390 2021-03-09T10:30:10.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:10.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:10.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00248e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:10.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.473+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:10.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.474+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00248e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:10.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.474+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:10.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.474+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:30:10.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.474+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:10.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.474+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:10.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:10.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:10.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:10.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:10.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:10.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:10.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00248e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:10.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00248e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:10.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:10.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:10.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:10.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:10.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a1a0 con 0x7f63e0018390 2021-03-09T10:30:10.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:10.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:10.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068e60 con 0x7f63e0018390 2021-03-09T10:30:10.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:10.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.475+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:10.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.477+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:30:10.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.477+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:10.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.477+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16804 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0036d20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:10.489 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:10.480+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46628 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:10.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:30:10.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.480+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:10.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.480+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:10.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.480+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:10.491 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:10.491 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:10.491 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:10.491 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:10.492 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:10.492 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:10.492 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:10.492 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:10.492 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:10.493 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:10.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63e0018390 2021-03-09T10:30:10.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.480+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:10.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.480+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:30:10.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.481+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:30:10.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.522+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:10.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:10.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:10.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:10.618+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:11.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:11.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:11.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:12.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:12.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:12.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:12.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:13.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:13.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:13.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:13.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:14.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:14.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:14.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:14.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:15.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:15.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:15.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16804 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0036d20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:15.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16804 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0036d20).stop 2021-03-09T10:30:15.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:15.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:15.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:15.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:15.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:30:15.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:15.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:15.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:15.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:15.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:15.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:15.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:15.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c320 con 0x7f63d8003400 2021-03-09T10:30:15.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:15.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8009130 2021-03-09T10:30:15.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:15.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:30:15.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.480+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:15.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.481+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:15.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.481+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:15.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.481+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:15.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.481+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:15.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.481+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:15.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.481+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:30:15.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.481+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:15.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.481+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:15.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.482+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:15.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.482+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:15.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.482+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:15.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:15.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:15.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:15.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:15.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:15.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:15.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:15.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:15.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:15.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00015a0 con 0x7f63d8003840 2021-03-09T10:30:15.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:15.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:15.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a1a0 con 0x7f63d8003840 2021-03-09T10:30:15.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:15.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.483+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:15.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.484+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:30:15.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.484+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:15.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.484+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16826 cs=0 l=1 rev1=1 rx=0x7f63f00216e0 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:15.496 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:15.486+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46669 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:15.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041eb0 con 0x7f63d8003840 2021-03-09T10:30:15.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.487+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:15.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.487+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:15.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.487+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:15.498 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:15.498 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:15.498 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:15.498 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:15.498 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:15.499 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:15.499 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:15.499 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:15.499 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:15.500 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:15.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.487+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041eb0 con 0x7f63d8003840 2021-03-09T10:30:15.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.487+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:15.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.487+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:30:15.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.488+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:30:15.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.523+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:15.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.618+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:15.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:15.618+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:16.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:16.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:16.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:16.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:17.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:17.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:17.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:17.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:18.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:18.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:18.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:18.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:19.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:19.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:19.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:19.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:20.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:20.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:20.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16826 cs=0 l=1 rev1=1 rx=0x7f63f00216e0 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:20.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16826 cs=0 l=1 rev1=1 rx=0x7f63f00216e0 tx=0x7f63f000df50).stop 2021-03-09T10:30:20.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:20.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:20.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:20.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=16826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:20.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:30:20.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:20.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:20.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:20.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:20.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:20.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:20.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:20.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011bd0 con 0x7f63d8003400 2021-03-09T10:30:20.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:20.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ab30 con 0x7f63d8009130 2021-03-09T10:30:20.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:20.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63e0018390 2021-03-09T10:30:20.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:20.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:20.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:20.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.488+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:20.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.489+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:20.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.489+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:20.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.489+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:30:20.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.489+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:20.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.489+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:20.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.489+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:20.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.489+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:20.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.489+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:20.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:20.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:20.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:20.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:20.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:20.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:20.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:20.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:20.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:20.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cb00 con 0x7f63e0018390 2021-03-09T10:30:20.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:20.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:20.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00015a0 con 0x7f63e0018390 2021-03-09T10:30:20.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:20.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:20.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16838 cs=0 l=1 rev1=1 rx=0x7f63f00428a0 tx=0x7f63f0066f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:20.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:30:20.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.490+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:20.504 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:20.493+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46707 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:20.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.495+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041eb0 con 0x7f63e0018390 2021-03-09T10:30:20.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.495+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:20.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.495+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:20.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.495+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:20.506 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:20.506 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:20.506 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:20.506 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:20.507 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:20.507 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:20.507 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:20.507 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:20.507 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:20.508 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:20.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.498+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041eb0 con 0x7f63e0018390 2021-03-09T10:30:20.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.498+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:20.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.498+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:30:20.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63e0018390 2021-03-09T10:30:20.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.524+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:20.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:20.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:20.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:20.619+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:21.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:21.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:21.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:21.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:22.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:22.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:22.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:22.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:23.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:23.619+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:23.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:23.619+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:24.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:24.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:24.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:24.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:25.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:25.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:25.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16838 cs=0 l=1 rev1=1 rx=0x7f63f00428a0 tx=0x7f63f0066f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:25.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16838 cs=0 l=1 rev1=1 rx=0x7f63f00428a0 tx=0x7f63f0066f00).stop 2021-03-09T10:30:25.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:25.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:25.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:25.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:25.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:30:25.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:25.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:25.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:25.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:25.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:25.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:25.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:25.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010850 con 0x7f63d8003400 2021-03-09T10:30:25.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:25.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:30:25.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:25.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b550 con 0x7f63d8003840 2021-03-09T10:30:25.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:25.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:25.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:25.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:25.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.494+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:25.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.495+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:25.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:30:25.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:25.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:25.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:25.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:25.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:25.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:25.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:25.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:25.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.496+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:25.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:25.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:25.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:25.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:25.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:25.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065370 con 0x7f63d8003840 2021-03-09T10:30:25.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:25.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:25.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cb00 con 0x7f63d8003840 2021-03-09T10:30:25.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:25.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:25.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:30:25.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:25.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.497+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 secure :-1 s=READY pgs=16852 cs=0 l=1 rev1=1 rx=0x7f63f0029660 tx=0x7f63f0036e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:25.509 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:25.500+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46747 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:25.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.501+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007c560 con 0x7f63d8003840 2021-03-09T10:30:25.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.501+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:25.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.501+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:25.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.501+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:25.511 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:25.511 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:25.511 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:25.511 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:25.511 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:25.512 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:25.512 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:25.512 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:25.512 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:25.512 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:25.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.503+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:30:25.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.503+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:25.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.503+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:30:25.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.504+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007c350 con 0x7f63d8003840 2021-03-09T10:30:25.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.525+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:25.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:25.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:25.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:26.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:26.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:26.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:26.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:27.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:27.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:27.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:27.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:28.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:28.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:28.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:28.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:29.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:29.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:29.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:29.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:30.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:30.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:30.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 secure :-1 s=READY pgs=16852 cs=0 l=1 rev1=1 rx=0x7f63f0029660 tx=0x7f63f0036e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:30.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 secure :-1 s=READY pgs=16852 cs=0 l=1 rev1=1 rx=0x7f63f0029660 tx=0x7f63f0036e00).stop 2021-03-09T10:30:30.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:30.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:30.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00114d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:30.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 unknown :-1 s=CLOSED pgs=16852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:30.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:30:30.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00246a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:30.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:30.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:30.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:30.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:30.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:30.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:30.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb70 con 0x7f63d8003400 2021-03-09T10:30:30.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:30.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e4b0 con 0x7f63d8009130 2021-03-09T10:30:30.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:30.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022a70 con 0x7f63e0018390 2021-03-09T10:30:30.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:30.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00246a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:30.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.500+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00246a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:30.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.501+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:30.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.501+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:30.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.501+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:30.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.501+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:30:30.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.501+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:30.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.501+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:30.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.501+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:30.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.501+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:30.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.501+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:30.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:30.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:30.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:30.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00246a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:30.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00246a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:30.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:30.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:30.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:30.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:30.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a1a0 con 0x7f63e0018390 2021-03-09T10:30:30.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:30.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:30.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065370 con 0x7f63e0018390 2021-03-09T10:30:30.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:30.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.502+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:30.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.503+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:30:30.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.503+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16858 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:30.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.503+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:30.516 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:30.506+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46785 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:30.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.508+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007a430 con 0x7f63e0018390 2021-03-09T10:30:30.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.508+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:30.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.508+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:30.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.508+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:30.518 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:30.518 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:30.518 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:30.518 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:30.518 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:30.519 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:30.519 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:30.519 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:30.519 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:30.520 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.510+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63e0018390 2021-03-09T10:30:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.510+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.510+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:30:30.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00813a0 con 0x7f63e0018390 2021-03-09T10:30:30.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.526+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:30.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:30.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:30.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:30.620+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:31.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:31.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:31.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:31.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:32.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:32.620+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:32.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:32.620+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:33.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:33.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:33.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:33.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:34.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:34.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:34.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:34.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:35.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:30:35.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:35.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16858 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:35.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16858 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f004db20).stop 2021-03-09T10:30:35.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:35.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:35.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:35.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:35.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:30:35.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:35.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:35.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:35.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:35.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:35.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:35.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:35.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:30:35.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:35.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023c70 con 0x7f63d8009130 2021-03-09T10:30:35.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:35.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00108a0 con 0x7f63d8003840 2021-03-09T10:30:35.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:35.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:35.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.507+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:35.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:35.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.508+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:35.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.509+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:35.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.509+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:30:35.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.509+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:35.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.509+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:35.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.509+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:35.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.509+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:35.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.509+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:35.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:35.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:35.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:35.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:35.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:35.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:35.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:35.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:35.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:35.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006df90 con 0x7f63d8003840 2021-03-09T10:30:35.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:35.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:35.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a1a0 con 0x7f63d8003840 2021-03-09T10:30:35.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:35.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:35.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16877 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f0056f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:35.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:30:35.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.510+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:35.523 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:35.513+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46826 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:35.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.515+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040640 con 0x7f63d8003840 2021-03-09T10:30:35.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.515+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:35.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.515+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:35.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.515+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:35.525 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:35.526 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:35.526 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:35.526 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:35.526 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:35.526 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:35.527 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:35.527 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:35.527 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:35.527 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:35.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.518+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8003840 2021-03-09T10:30:35.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.518+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:35.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.518+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:30:35.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.519+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00298c0 con 0x7f63d8003840 2021-03-09T10:30:35.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.527+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:35.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:35.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:35.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:36.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:36.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:36.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:36.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:37.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:37.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:37.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:37.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:38.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:38.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:38.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:38.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:39.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:39.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:39.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:39.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:40.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:40.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:40.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16877 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f0056f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:40.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16877 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f0056f70).stop 2021-03-09T10:30:40.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:40.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:40.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=16877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:30:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:40.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:40.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:40.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029e30 con 0x7f63d8003400 2021-03-09T10:30:40.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:40.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017000 con 0x7f63d8009130 2021-03-09T10:30:40.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:40.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:30:40.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:40.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:40.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:40.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:40.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:40.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:40.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:30:40.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:40.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:40.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:40.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:40.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:40.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:40.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:40.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:40.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:40.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:40.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:30:40.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:30:40.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:40.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:40.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cb00 con 0x7f63e0018390 2021-03-09T10:30:40.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:40.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:40.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.515+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010cf0 con 0x7f63e0018390 2021-03-09T10:30:40.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.516+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:30:40.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.516+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:40.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.516+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:30:40.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.516+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:40.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.516+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16887 cs=0 l=1 rev1=1 rx=0x7f63fc05d070 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:40.532 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:40.520+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46864 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:40.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:30:40.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.520+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:40.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.520+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:40.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.520+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:40.534 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:40.534 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:40.534 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:40.534 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:40.534 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:40.535 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:40.536 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.521+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:30:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.521+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.521+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:30:40.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:30:40.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.528+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:40.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.621+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:40.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.621+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:40.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:40.621+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:41.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:41.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:41.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:41.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:42.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:42.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:43.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:43.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:43.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:43.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:44.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:44.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:44.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:44.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:45.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:45.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:45.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16887 cs=0 l=1 rev1=1 rx=0x7f63fc05d070 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:45.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16887 cs=0 l=1 rev1=1 rx=0x7f63fc05d070 tx=0x7f63fc04a6d0).stop 2021-03-09T10:30:45.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:45.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:45.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:45.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:45.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:30:45.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:45.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:45.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:45.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:30:45.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:45.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012c70 con 0x7f63d8009130 2021-03-09T10:30:45.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:45.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bb80 con 0x7f63d8003840 2021-03-09T10:30:45.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:45.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:45.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.520+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:45.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:45.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:45.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:45.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:30:45.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:45.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:45.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:45.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:45.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:45.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:45.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:45.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:45.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:45.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:45.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:45.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017950 con 0x7f63d8003840 2021-03-09T10:30:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:45.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006df90 con 0x7f63d8003840 2021-03-09T10:30:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.522+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.522+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.522+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 secure :-1 s=READY pgs=16899 cs=0 l=1 rev1=1 rx=0x7f63f0040800 tx=0x7f63f007ef60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:45.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.522+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:30:45.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.522+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:45.535 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:45.527+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46905 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007bc80 con 0x7f63d8003840 2021-03-09T10:30:45.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.528+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:45.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.528+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:45.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.528+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:45.537 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:45.537 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:45.537 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:45.538 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:45.538 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:45.538 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:45.538 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:45.538 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:45.539 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8003840 2021-03-09T10:30:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.528+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:45.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.528+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:30:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.529+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:45.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:30:45.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:45.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:45.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:46.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:46.622+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:46.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:46.622+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:47.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:47.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:47.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:47.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:48.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:48.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:48.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:48.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:49.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:49.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:49.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:49.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:50.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:50.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:50.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 secure :-1 s=READY pgs=16899 cs=0 l=1 rev1=1 rx=0x7f63f0040800 tx=0x7f63f007ef60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:50.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 secure :-1 s=READY pgs=16899 cs=0 l=1 rev1=1 rx=0x7f63f0040800 tx=0x7f63f007ef60).stop 2021-03-09T10:30:50.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:50.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:50.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:50.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027e40 unknown :-1 s=CLOSED pgs=16899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:50.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:30:50.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:50.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:50.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:50.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:50.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:50.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:50.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:50.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:30:50.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:50.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b980 con 0x7f63d8009130 2021-03-09T10:30:50.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:50.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008f30 con 0x7f63e0018390 2021-03-09T10:30:50.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:50.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:50.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:50.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:50.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.528+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:50.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:50.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:30:50.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:50.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:50.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:50.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:50.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:50.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:50.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:50.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:50.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:50.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:50.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:30:50.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:30:50.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:50.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:50.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006df90 con 0x7f63e0018390 2021-03-09T10:30:50.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:50.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:50.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cb00 con 0x7f63e0018390 2021-03-09T10:30:50.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:30:50.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.529+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:50.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.530+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:50.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.530+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:30:50.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.530+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:50.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.530+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16912 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc03fea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:50.545 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:50.535+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46943 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:50.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.535+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:30:50.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.535+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:50.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.535+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:50.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.535+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:50.547 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:50.547 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:50.547 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:50.548 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:50.548 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:50.548 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:50.548 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:50.548 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:50.549 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:50.549 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:50.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.535+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0711d0 con 0x7f63e0018390 2021-03-09T10:30:50.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.535+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:50.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.535+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:30:50.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.536+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f63e0018390 2021-03-09T10:30:50.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:50.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:50.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:50.623+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:51.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:51.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:51.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:51.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:52.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:52.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:52.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:52.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:53.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:53.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:53.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:53.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:54.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:54.623+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:54.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:54.623+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:55.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.531+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:30:55.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:30:55.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:30:55.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16912 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc03fea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:30:55.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16912 cs=0 l=1 rev1=1 rx=0x7f63fc055040 tx=0x7f63fc03fea0).stop 2021-03-09T10:30:55.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:55.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:30:55.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:30:55.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:30:55.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:30:55.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:55.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:30:55.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.535+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:55.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:30:55.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:30:55.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:30:55.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:55.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ad30 con 0x7f63d8003400 2021-03-09T10:30:55.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:55.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:30:55.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:30:55.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:30:55.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:30:55.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:30:55.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:55.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:30:55.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.536+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:30:55.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.537+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:30:55.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.537+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:30:55.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.537+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:30:55.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.537+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:30:55.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.537+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:30:55.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.537+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:30:55.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.537+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:30:55.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:30:55.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:55.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:55.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:30:55.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:30:55.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:30:55.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:30:55.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:30:55.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:55.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0069550 con 0x7f63d8003840 2021-03-09T10:30:55.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:30:55.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:30:55.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017950 con 0x7f63d8003840 2021-03-09T10:30:55.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:30:55.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:55.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16936 cs=0 l=1 rev1=1 rx=0x7f63f007a820 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:30:55.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:30:55.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.539+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:30:55.550 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:30:55.542+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 46984 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:30:55.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.542+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:30:55.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.542+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:30:55.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.543+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:30:55.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.543+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:30:55.552 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:30:55.552 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:30:55.552 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:55.552 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:30:55.553 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:30:55.553 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:30:55.553 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:30:55.553 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:30:55.553 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:30:55.554 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:30:55.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.544+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:30:55.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.544+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:30:55.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.544+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:30:55.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.545+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:30:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:55.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:55.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:56.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:56.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:56.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:56.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:57.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:57.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:57.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:57.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:58.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:58.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:58.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:58.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:30:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:59.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:30:59.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:30:59.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:00.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.532+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:00.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:00.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:00.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16936 cs=0 l=1 rev1=1 rx=0x7f63f007a820 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:00.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16936 cs=0 l=1 rev1=1 rx=0x7f63f007a820 tx=0x7f63f0013b80).stop 2021-03-09T10:31:00.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:00.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:00.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:00.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=16936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:00.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:31:00.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:00.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:00.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:00.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:00.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:00.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:00.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:00.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb70 con 0x7f63d8003400 2021-03-09T10:31:00.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:00.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008f30 con 0x7f63d8009130 2021-03-09T10:31:00.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:00.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e0018390 2021-03-09T10:31:00.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:00.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:00.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:00.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:00.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:31:00.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:00.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.543+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:00.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.544+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:00.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.544+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:00.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.544+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:00.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.544+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:00.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.544+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:00.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:00.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:00.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:00.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:00.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:00.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:31:00.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:31:00.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:00.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:00.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000cb00 con 0x7f63e0018390 2021-03-09T10:31:00.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:00.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:00.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0069550 con 0x7f63e0018390 2021-03-09T10:31:00.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:31:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:31:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:00.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.545+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16945 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00886f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:00.559 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:00.551+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47022 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:00.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63e0018390 2021-03-09T10:31:00.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.551+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:00.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.551+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:00.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.551+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:00.561 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:00.561 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:00.561 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:00.561 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:00.562 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:00.562 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:00.562 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:00.562 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:00.563 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:00.563 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:00.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.553+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63e0018390 2021-03-09T10:31:00.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.553+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.553+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:31:00.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:31:00.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:00.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:00.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:00.624+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:01.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:01.624+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:01.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:01.624+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:02.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:02.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:02.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:02.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:03.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:03.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:03.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:03.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:04.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:04.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:04.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:04.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:05.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.533+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:05.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:05.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:05.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16945 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00886f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:05.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16945 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f00886f0).stop 2021-03-09T10:31:05.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:05.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:05.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:05.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:05.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:31:05.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:05.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:05.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:05.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:05.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:05.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:05.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:05.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ab30 con 0x7f63d8003400 2021-03-09T10:31:05.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:05.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023c70 con 0x7f63d8009130 2021-03-09T10:31:05.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:05.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8003840 2021-03-09T10:31:05.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:05.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:05.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:05.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:05.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:05.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:05.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:31:05.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:05.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:05.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:05.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:05.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.552+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:05.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:05.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:05.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:05.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:05.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:05.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:31:05.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:31:05.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:05.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:05.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022f60 con 0x7f63d8003840 2021-03-09T10:31:05.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:05.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:05.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000cb00 con 0x7f63d8003840 2021-03-09T10:31:05.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:31:05.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.553+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:05.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.554+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16955 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0066460).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:05.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.554+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:31:05.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.554+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:05.566 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:05.559+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47063 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:05.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.559+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027e00 con 0x7f63d8003840 2021-03-09T10:31:05.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.559+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:05.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.559+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:05.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.559+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:05.568 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:05.568 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:05.569 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:05.569 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:05.569 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:05.569 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:05.569 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:05.570 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:05.570 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:05.570 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:05.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.559+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027c20 con 0x7f63d8003840 2021-03-09T10:31:05.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.559+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:05.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.560+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:31:05.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.561+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8003840 2021-03-09T10:31:05.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:05.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:05.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:06.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:06.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:06.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:06.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:07.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:07.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:07.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:08.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:08.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:08.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:08.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:09.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:09.625+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:09.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:09.625+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:10.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.534+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:10.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:10.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:10.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16955 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0066460).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:10.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=16955 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0066460).stop 2021-03-09T10:31:10.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:10.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:10.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:10.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=16955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:10.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:31:10.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:10.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:10.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:10.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:10.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:10.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:10.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:10.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029df0 con 0x7f63d8003400 2021-03-09T10:31:10.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:10.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:31:10.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:10.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ad30 con 0x7f63e0018390 2021-03-09T10:31:10.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.559+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:10.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.560+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:10.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.560+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:10.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.560+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:10.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.560+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:10.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.561+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:10.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.561+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:31:10.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.561+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:10.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.561+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.561+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.561+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:10.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.561+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:10.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:10.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:31:10.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:31:10.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:10.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:10.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007ad40 con 0x7f63e0018390 2021-03-09T10:31:10.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:10.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:10.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022f60 con 0x7f63e0018390 2021-03-09T10:31:10.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:31:10.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:10.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:31:10.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:10.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.562+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16975 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f0028610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:10.574 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:10.566+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47101 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:10.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.567+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00287e0 con 0x7f63e0018390 2021-03-09T10:31:10.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.567+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:10.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.567+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:10.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.567+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:10.576 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:10.576 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:10.576 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:10.576 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:10.576 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:10.577 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:10.577 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:10.577 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:10.577 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:10.577 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:10.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63e0018390 2021-03-09T10:31:10.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.571+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:10.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.571+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:31:10.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.572+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63e0018390 2021-03-09T10:31:10.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:10.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:10.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:10.626+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:11.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:11.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:11.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:11.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:12.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:12.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:12.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:12.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:13.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:13.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:13.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:13.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:14.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:14.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:14.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:14.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:15.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.535+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:15.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:15.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:15.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16975 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f0028610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:15.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16975 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f0028610).stop 2021-03-09T10:31:15.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:15.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:15.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:15.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:15.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:31:15.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:15.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:15.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:15.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8003400 2021-03-09T10:31:15.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:15.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e320 con 0x7f63d8009130 2021-03-09T10:31:15.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:15.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8003840 2021-03-09T10:31:15.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:15.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.566+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:15.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.567+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:15.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.567+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:15.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.567+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:15.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.567+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:15.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.567+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:31:15.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.567+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:15.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.567+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:15.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.568+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:15.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.568+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:15.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.568+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:15.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:15.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:15.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:15.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:15.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:15.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:31:15.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:31:15.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:15.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:15.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002abf0 con 0x7f63d8003840 2021-03-09T10:31:15.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:15.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:15.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007ad40 con 0x7f63d8003840 2021-03-09T10:31:15.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:31:15.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.569+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:15.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.570+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b840 secure :-1 s=READY pgs=16989 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:15.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.570+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:31:15.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.570+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:15.583 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:15.574+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47142 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:15.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.575+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8003840 2021-03-09T10:31:15.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.575+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:15.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.575+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:15.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.575+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:15.584 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:15.584 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:15.585 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:15.585 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:15.585 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:15.585 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:15.586 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:15.586 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:15.586 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:15.586 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:15.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.577+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8003840 2021-03-09T10:31:15.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.577+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:15.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.577+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:31:15.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.578+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:31:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:15.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:15.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:16.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:16.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:16.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:16.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:17.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:17.626+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:17.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:17.626+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:18.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:18.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:18.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:18.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:19.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:19.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:19.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:19.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:20.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.536+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:20.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:20.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:20.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b840 secure :-1 s=READY pgs=16989 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:20.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b840 secure :-1 s=READY pgs=16989 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f004dd90).stop 2021-03-09T10:31:20.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:20.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:20.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007b840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:20.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b840 unknown :-1 s=CLOSED pgs=16989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:20.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:31:20.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:20.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:20.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:20.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:20.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:20.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:20.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:20.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb70 con 0x7f63d8003400 2021-03-09T10:31:20.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:20.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022910 con 0x7f63d8009130 2021-03-09T10:31:20.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021d40 con 0x7f63e0018390 2021-03-09T10:31:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:20.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:31:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:20.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.575+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:20.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:20.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:20.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:20.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:20.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:20.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:31:20.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:31:20.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:20.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:20.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00015a0 con 0x7f63e0018390 2021-03-09T10:31:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006df90 con 0x7f63e0018390 2021-03-09T10:31:20.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:31:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:31:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:20.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.576+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16994 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc04bbf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:20.590 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:20.582+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47180 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:20.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63e0018390 2021-03-09T10:31:20.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.582+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:20.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.582+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:20.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.582+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:20.592 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:20.592 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:20.592 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:20.592 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:20.593 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:20.593 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:20.593 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:20.593 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:20.593 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:20.594 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:20.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63e0018390 2021-03-09T10:31:20.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.582+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:20.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.582+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:31:20.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.583+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012240 con 0x7f63e0018390 2021-03-09T10:31:20.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:20.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:20.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:20.627+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:21.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:21.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:21.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:21.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:22.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:22.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:22.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:22.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:23.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:23.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:23.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:23.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:24.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:24.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:24.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:24.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:25.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.537+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:25.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:25.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:25.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16994 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc04bbf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:25.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=16994 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc04bbf0).stop 2021-03-09T10:31:25.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:25.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:25.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:25.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=16994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:25.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:31:25.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:25.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:25.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:25.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:25.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:25.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:25.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:25.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:31:25.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:25.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001730 con 0x7f63d8009130 2021-03-09T10:31:25.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:25.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dd50 con 0x7f63d8003840 2021-03-09T10:31:25.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:25.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.582+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:25.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:25.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:25.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:31:25.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:25.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:25.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:25.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:25.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:25.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:25.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.583+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:25.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:25.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:25.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:25.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e00245a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:25.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e00245a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:25.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:31:25.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:31:25.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:25.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:25.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:31:25.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:25.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:25.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002abf0 con 0x7f63d8003840 2021-03-09T10:31:25.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:31:25.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:25.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:31:25.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:25.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.584+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 secure :-1 s=READY pgs=17011 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0066380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:25.611 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:25.589+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47221 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:25.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036d80 con 0x7f63d8003840 2021-03-09T10:31:25.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.592+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:25.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.592+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:25.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.592+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:25.614 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:25.614 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:25.615 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:25.615 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:25.616 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:25.616 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:25.616 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:25.617 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:25.617 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:25.618 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:25.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.592+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0036d80 con 0x7f63d8003840 2021-03-09T10:31:25.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.592+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:25.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.592+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:31:25.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.595+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00882e0 con 0x7f63d8003840 2021-03-09T10:31:25.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:25.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:25.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:26.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:26.627+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:26.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:26.627+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:27.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:27.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:27.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:27.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:28.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:28.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:28.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:28.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:29.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:29.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:29.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:29.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:30.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.538+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:30.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:30.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:30.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 secure :-1 s=READY pgs=17011 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0066380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:30.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 secure :-1 s=READY pgs=17011 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0066380).stop 2021-03-09T10:31:30.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:30.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:30.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:30.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 unknown :-1 s=CLOSED pgs=17011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:30.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:31:30.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:30.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:30.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:30.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:30.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:30.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:30.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:30.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb70 con 0x7f63d8003400 2021-03-09T10:31:30.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:30.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064d50 con 0x7f63d8009130 2021-03-09T10:31:30.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:30.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63e0018390 2021-03-09T10:31:30.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:30.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:30.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.590+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:30.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.591+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:30.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.591+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:30.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.591+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:30.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.591+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:31:30.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.591+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:30.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.591+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:30.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.591+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:30.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.591+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:30.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.591+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:30.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:30.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:30.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:30.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:30.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:30.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:31:30.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:31:30.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:30.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:30.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c670 con 0x7f63e0018390 2021-03-09T10:31:30.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:30.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:30.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00015a0 con 0x7f63e0018390 2021-03-09T10:31:30.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:31:30.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.592+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:30.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.593+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:31:30.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.593+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:30.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.593+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17030 cs=0 l=1 rev1=1 rx=0x7f63fc0583a0 tx=0x7f63fc052f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:30.606 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:30.597+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47259 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:30.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.597+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058410 con 0x7f63e0018390 2021-03-09T10:31:30.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.597+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:30.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.597+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:30.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.597+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:30.607 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:30.608 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:30.609 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:30.609 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:30.609 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:30.609 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:30.609 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.598+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0581e0 con 0x7f63e0018390 2021-03-09T10:31:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.598+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.598+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:31:30.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f63e0018390 2021-03-09T10:31:30.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:30.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:30.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:30.628+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:31.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:31.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:32.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:32.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:32.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:32.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:33.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:33.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:33.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:33.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:34.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:34.628+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:34.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:34.628+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:35.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.539+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:35.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:35.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:35.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17030 cs=0 l=1 rev1=1 rx=0x7f63fc0583a0 tx=0x7f63fc052f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:35.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17030 cs=0 l=1 rev1=1 rx=0x7f63fc0583a0 tx=0x7f63fc052f70).stop 2021-03-09T10:31:35.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:35.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:35.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:35.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:35.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:31:35.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:35.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:35.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:35.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:35.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:35.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:35.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:35.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d3a0 con 0x7f63d8003400 2021-03-09T10:31:35.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:35.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cdf0 con 0x7f63d8009130 2021-03-09T10:31:35.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:35.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067570 con 0x7f63d8003840 2021-03-09T10:31:35.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:35.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:35.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:35.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:35.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.598+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:35.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.599+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:35.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:31:35.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:35.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:35.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:35.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:35.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:35.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:35.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:35.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:35.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:35.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:35.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:31:35.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:31:35.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:35.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:35.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce40 con 0x7f63d8003840 2021-03-09T10:31:35.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:35.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:35.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:31:35.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:31:35.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.600+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:35.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.601+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:31:35.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.601+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=17043 cs=0 l=1 rev1=1 rx=0x7f63e0028570 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:35.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.601+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:35.615 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:35.604+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47300 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:35.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63d8003840 2021-03-09T10:31:35.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.604+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:35.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.604+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:35.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.604+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:35.616 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:35.617 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:35.617 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:35.617 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:35.617 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:35.618 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:35.618 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:35.618 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:35.618 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:35.619 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:35.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.607+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63d8003840 2021-03-09T10:31:35.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.607+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:35.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.607+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:31:35.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:31:35.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:35.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:35.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:36.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:36.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:37.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:37.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:37.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:37.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:38.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:38.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:38.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:38.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:39.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:39.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:39.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:39.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:40.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.540+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=17043 cs=0 l=1 rev1=1 rx=0x7f63e0028570 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:40.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=17043 cs=0 l=1 rev1=1 rx=0x7f63e0028570 tx=0x7f63f004db20).stop 2021-03-09T10:31:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:40.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=17043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:40.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:31:40.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:40.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:40.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:40.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:40.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:40.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:40.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:40.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:31:40.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:40.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021be0 con 0x7f63d8009130 2021-03-09T10:31:40.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:40.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068470 con 0x7f63e0018390 2021-03-09T10:31:40.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:40.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:40.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:40.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:40.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:40.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:40.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:31:40.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:40.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:40.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:40.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:40.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:40.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:40.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:40.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:40.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:40.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:40.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:31:40.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:31:40.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:40.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:40.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T10:31:40.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:40.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:40.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c670 con 0x7f63e0018390 2021-03-09T10:31:40.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:31:40.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.606+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:40.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.607+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:31:40.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.607+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17051 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc051a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:40.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.607+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:40.629 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:40.611+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47338 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:40.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058000 con 0x7f63e0018390 2021-03-09T10:31:40.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.611+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:40.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.611+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:40.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.611+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:40.632 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:40.632 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:40.632 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:40.632 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:40.633 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:40.633 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:40.633 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:40.633 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:40.634 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:40.634 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:40.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065e60 con 0x7f63e0018390 2021-03-09T10:31:40.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.611+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:40.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.611+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:31:40.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.612+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:31:40.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.629+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:40.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.629+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:40.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:40.629+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:41.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:41.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:41.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:41.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:42.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:42.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:42.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:42.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:43.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:43.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:43.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:43.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:44.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:44.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:44.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:44.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:45.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.541+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:45.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:45.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:45.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17051 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc051a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17051 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc051a30).stop 2021-03-09T10:31:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:45.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:45.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:45.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:31:45.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:45.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:45.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:45.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003400 2021-03-09T10:31:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb70 con 0x7f63d8009130 2021-03-09T10:31:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:45.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d3a0 con 0x7f63d8003840 2021-03-09T10:31:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:45.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.611+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.612+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.612+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:31:45.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.612+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:45.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.612+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:45.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.612+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:45.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.612+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:45.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.612+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:45.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:45.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:45.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:45.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:31:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:31:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:45.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:31:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce40 con 0x7f63d8003840 2021-03-09T10:31:45.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:31:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069030 secure :-1 s=READY pgs=17065 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f00882e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:31:45.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.613+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:45.630 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:45.618+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47379 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0088ef0 con 0x7f63d8003840 2021-03-09T10:31:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.618+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.618+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.618+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:45.631 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:45.632 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:45.633 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:45.633 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:31:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.618+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:45.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.618+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:31:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.619+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:31:45.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:45.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:46.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:46.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:46.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:46.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:47.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:47.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:47.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:47.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:48.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:48.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:48.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:48.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:49.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:49.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:49.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:49.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:50.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.542+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:50.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0069030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:31:50.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0069030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:50.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069030 secure :-1 s=READY pgs=17065 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f00882e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:50.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069030 secure :-1 s=READY pgs=17065 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f00882e0).stop 2021-03-09T10:31:50.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:50.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:50.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0069030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:50.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069030 unknown :-1 s=CLOSED pgs=17065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:50.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:31:50.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:50.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:50.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:50.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:50.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:50.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:50.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:50.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:31:50.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:50.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:31:50.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:50.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008dd0 con 0x7f63e0018390 2021-03-09T10:31:50.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:50.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:50.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:50.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:50.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.618+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:50.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:50.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:31:50.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:50.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:50.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:50.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:50.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:50.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:50.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:50.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:50.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:50.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:50.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:31:50.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:31:50.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:50.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:50.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00254c0 con 0x7f63e0018390 2021-03-09T10:31:50.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:50.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:50.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.619+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63e0018390 2021-03-09T10:31:50.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.620+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:31:50.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.620+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:50.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.620+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17078 cs=0 l=1 rev1=1 rx=0x7f63f0027820 tx=0x7f63f004d1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:50.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.620+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:31:50.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.620+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:50.645 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:50.626+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47417 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:50.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.626+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63e0018390 2021-03-09T10:31:50.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.626+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:50.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.626+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:50.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.626+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:50.647 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:50.647 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:50.647 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:50.648 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:50.648 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:50.648 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:50.648 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:50.648 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:50.649 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:50.649 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:50.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.626+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027430 con 0x7f63e0018390 2021-03-09T10:31:50.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.626+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:50.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.626+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:31:50.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63e0018390 2021-03-09T10:31:50.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:50.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:50.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:50.630+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:51.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:51.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:51.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:51.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:52.630+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:52.630+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:53.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:53.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:53.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:53.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:54.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:54.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:54.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:54.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:55.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.543+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:31:55.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:31:55.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:31:55.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17078 cs=0 l=1 rev1=1 rx=0x7f63f0027820 tx=0x7f63f004d1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:31:55.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17078 cs=0 l=1 rev1=1 rx=0x7f63f0027820 tx=0x7f63f004d1f0).stop 2021-03-09T10:31:55.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:55.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:31:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:31:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:31:55.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:31:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00152a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:31:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:31:55.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:31:55.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:31:55.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:55.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dd50 con 0x7f63d8003400 2021-03-09T10:31:55.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028da0 con 0x7f63d8009130 2021-03-09T10:31:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:31:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d0b0 con 0x7f63d8003840 2021-03-09T10:31:55.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:31:55.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:31:55.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.626+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:55.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.627+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:31:55.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.627+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:31:55.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.627+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00152a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:31:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.627+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:31:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.628+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:31:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.628+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:31:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:31:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:31:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.629+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:31:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:31:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:55.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:31:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:31:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:31:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:31:55.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:31:55.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:55.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:31:55.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:31:55.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:31:55.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00254c0 con 0x7f63d8003840 2021-03-09T10:31:55.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:31:55.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:55.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00152a0 secure :-1 s=READY pgs=17103 cs=0 l=1 rev1=1 rx=0x7f63f0060070 tx=0x7f63f007f530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:31:55.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:31:55.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.630+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:31:55.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:55.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:55.643 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:31:55.634+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47458 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:31:55.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.634+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027460 con 0x7f63d8003840 2021-03-09T10:31:55.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.634+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:31:55.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.634+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:31:55.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.634+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:31:55.645 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:31:55.645 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:31:55.645 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:55.645 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:31:55.646 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:31:55.646 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:31:55.646 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:31:55.646 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:31:55.647 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:31:55.647 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:31:55.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.638+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007ec20 con 0x7f63d8003840 2021-03-09T10:31:55.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.638+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:31:55.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.638+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:31:55.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:55.638+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042320 con 0x7f63d8003840 2021-03-09T10:31:56.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:56.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:56.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:56.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:57.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:57.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:57.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:57.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:58.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:58.631+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:58.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:58.631+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:31:59.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:59.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:31:59.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:31:59.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:00.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.544+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:00.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:00.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:00.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.632+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:00.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:00.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00152a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:00.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00152a0 secure :-1 s=READY pgs=17103 cs=0 l=1 rev1=1 rx=0x7f63f0060070 tx=0x7f63f007f530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:00.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00152a0 secure :-1 s=READY pgs=17103 cs=0 l=1 rev1=1 rx=0x7f63f0060070 tx=0x7f63f007f530).stop 2021-03-09T10:32:00.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:00.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:00.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00152a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00152a0 unknown :-1 s=CLOSED pgs=17103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:32:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:00.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:00.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:00.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:00.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:00.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:00.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d3a0 con 0x7f63d8003400 2021-03-09T10:32:00.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:00.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:32:00.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:00.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068470 con 0x7f63e0018390 2021-03-09T10:32:00.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:00.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:00.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:00.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:00.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:00.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:00.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:32:00.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:00.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.635+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:00.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.636+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:00.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.636+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:00.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.636+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:00.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:00.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:00.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:00.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:00.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:00.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:32:00.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:32:00.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:00.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:00.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006def0 con 0x7f63e0018390 2021-03-09T10:32:00.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:00.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:00.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63e0018390 2021-03-09T10:32:00.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:32:00.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:00.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:32:00.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:00.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17113 cs=0 l=1 rev1=1 rx=0x7f63f002d310 tx=0x7f63f0032f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:00.649 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:00.641+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47496 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:00.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040bc0 con 0x7f63e0018390 2021-03-09T10:32:00.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.642+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:00.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.642+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:00.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.642+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:00.651 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:00.651 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:00.651 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:00.651 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:00.652 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:00.652 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:00.652 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:00.652 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:00.652 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:00.653 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:00.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.643+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040bc0 con 0x7f63e0018390 2021-03-09T10:32:00.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.643+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:00.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.643+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:32:00.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:00.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0088350 con 0x7f63e0018390 2021-03-09T10:32:01.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:01.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:01.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:01.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:02.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:02.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:02.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:02.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:03.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:03.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:03.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:03.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:04.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:04.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:04.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:04.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:05.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.545+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:05.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:05.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:05.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:05.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:05.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17113 cs=0 l=1 rev1=1 rx=0x7f63f002d310 tx=0x7f63f0032f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:05.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17113 cs=0 l=1 rev1=1 rx=0x7f63f002d310 tx=0x7f63f0032f80).stop 2021-03-09T10:32:05.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:05.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:05.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:05.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:05.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:32:05.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:05.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:05.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:05.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:05.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:05.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:05.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:05.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c3e0 con 0x7f63d8003400 2021-03-09T10:32:05.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:05.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028da0 con 0x7f63d8009130 2021-03-09T10:32:05.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:05.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069410 con 0x7f63d8003840 2021-03-09T10:32:05.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:05.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:05.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:05.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:05.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.642+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:05.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.643+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:05.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.643+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:32:05.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.643+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:05.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.643+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:05.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.643+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:05.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.643+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:05.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.643+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:05.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:05.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:05.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:05.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:05.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:05.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:32:05.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:32:05.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:05.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:05.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007ad40 con 0x7f63d8003840 2021-03-09T10:32:05.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:05.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:05.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006def0 con 0x7f63d8003840 2021-03-09T10:32:05.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:32:05.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.644+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:05.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.645+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:32:05.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.645+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17126 cs=0 l=1 rev1=1 rx=0x7f63f0040fb0 tx=0x7f63f004c2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:05.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.645+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:05.657 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:05.649+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47537 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:05.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.649+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028480 con 0x7f63d8003840 2021-03-09T10:32:05.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.649+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:05.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.649+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:05.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.649+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:05.659 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:05.660 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:05.660 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:05.660 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:05.660 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:05.660 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:05.661 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:05.661 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:05.661 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:05.661 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:05.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.651+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63d8003840 2021-03-09T10:32:05.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.651+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:05.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.651+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:32:05.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:05.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0042bd0 con 0x7f63d8003840 2021-03-09T10:32:06.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:06.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:06.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:06.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:07.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:07.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:07.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:07.632+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:08.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:08.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:08.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:08.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:09.632+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:09.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:09.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:10.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.546+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:10.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:10.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:10.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.634+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:10.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:10.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:10.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17126 cs=0 l=1 rev1=1 rx=0x7f63f0040fb0 tx=0x7f63f004c2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:10.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17126 cs=0 l=1 rev1=1 rx=0x7f63f0040fb0 tx=0x7f63f004c2d0).stop 2021-03-09T10:32:10.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:10.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:10.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:10.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=CLOSED pgs=17126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:10.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:32:10.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:10.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:10.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:10.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:10.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:10.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:10.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:10.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:32:10.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:10.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028fa0 con 0x7f63d8009130 2021-03-09T10:32:10.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:10.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:32:10.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:10.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:10.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:10.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:32:10.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:10.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:10.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:10.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:10.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.650+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:10.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.651+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:10.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.651+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:10.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.651+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:10.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:10.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:10.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:10.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:10.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:10.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:32:10.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:32:10.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:10.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:10.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a710 con 0x7f63e0018390 2021-03-09T10:32:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T10:32:10.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:32:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:32:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17138 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc06ffe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.652+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:10.665 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:10.656+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47575 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:10.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fea0 con 0x7f63e0018390 2021-03-09T10:32:10.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.656+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:10.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.656+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:10.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.656+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:10.667 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:10.667 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:10.667 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:10.667 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:10.668 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:10.669 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.656+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051bd0 con 0x7f63e0018390 2021-03-09T10:32:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.656+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:10.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.656+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:32:10.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:10.657+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:32:11.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:11.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:11.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:11.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:12.633+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:12.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:12.633+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:13.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:13.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:13.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:13.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:14.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:14.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:14.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:14.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:15.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.547+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:15.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:15.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:15.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:15.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:15.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17138 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc06ffe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:15.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17138 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc06ffe0).stop 2021-03-09T10:32:15.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:15.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:15.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:15.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:15.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:32:15.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:15.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:15.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:15.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:15.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:15.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:15.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:15.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027cb0 con 0x7f63d8003400 2021-03-09T10:32:15.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:15.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:32:15.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:15.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016330 con 0x7f63d8003840 2021-03-09T10:32:15.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:15.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:15.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:15.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:15.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:15.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:15.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:32:15.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:15.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:15.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:15.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:15.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.657+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:15.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:15.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:15.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:15.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:15.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:15.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:32:15.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:32:15.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:15.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:15.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:32:15.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:15.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:15.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007ad40 con 0x7f63d8003840 2021-03-09T10:32:15.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:32:15.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.658+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:15.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.659+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:32:15.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.659+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:15.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.659+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17147 cs=0 l=1 rev1=1 rx=0x7f63f0088b50 tx=0x7f63f0041f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:15.673 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:15.663+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47616 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:15.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027430 con 0x7f63d8003840 2021-03-09T10:32:15.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.664+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:15.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.664+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:15.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.664+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:15.675 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:15.675 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:15.675 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:15.675 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:15.676 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:15.677 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:15.677 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:15.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:32:15.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.664+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:15.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.664+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:32:15.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:15.665+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63d8003840 2021-03-09T10:32:16.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:16.634+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:16.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:16.634+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:17.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:17.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:17.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:17.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:18.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:18.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:18.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:18.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:19.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:19.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:19.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:20.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.548+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:20.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:20.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:20.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.635+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:20.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:20.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:20.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17147 cs=0 l=1 rev1=1 rx=0x7f63f0088b50 tx=0x7f63f0041f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:20.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17147 cs=0 l=1 rev1=1 rx=0x7f63f0088b50 tx=0x7f63f0041f00).stop 2021-03-09T10:32:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=CLOSED pgs=17147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:32:20.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:20.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:20.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:20.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:20.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:20.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:20.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:20.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00689c0 con 0x7f63d8003400 2021-03-09T10:32:20.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:20.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bd80 con 0x7f63d8009130 2021-03-09T10:32:20.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:20.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c8f0 con 0x7f63e0018390 2021-03-09T10:32:20.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:20.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:20.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:32:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.664+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:32:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:32:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:20.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0069550 con 0x7f63e0018390 2021-03-09T10:32:20.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:20.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:20.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a710 con 0x7f63e0018390 2021-03-09T10:32:20.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:32:20.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.665+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:20.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.666+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:32:20.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.666+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17161 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc03ff10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:20.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.666+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:20.678 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:20.672+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47654 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:20.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:32:20.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.672+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:20.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.672+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:20.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.672+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:20.680 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:20.680 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:20.681 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:20.681 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:20.681 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:20.681 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:20.681 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:20.682 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:20.682 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:20.682 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:20.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0711d0 con 0x7f63e0018390 2021-03-09T10:32:20.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.672+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:20.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.672+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:32:20.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:20.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:32:21.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:21.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:21.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:21.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:22.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:22.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:22.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:22.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:23.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:23.635+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:23.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:23.635+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:24.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:24.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:24.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:24.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:25.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.549+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:25.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:25.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:25.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:25.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:25.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17161 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc03ff10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:25.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17161 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc03ff10).stop 2021-03-09T10:32:25.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:25.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:25.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:25.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:25.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:32:25.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:25.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:25.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:25.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:25.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:25.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:25.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:25.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:32:25.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:25.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016330 con 0x7f63d8009130 2021-03-09T10:32:25.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:25.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025cb0 con 0x7f63d8003840 2021-03-09T10:32:25.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:25.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:25.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:25.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.672+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:25.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.673+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:25.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.673+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:25.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.673+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:32:25.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.673+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:25.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.673+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:25.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.673+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:25.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.673+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:25.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.673+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:25.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:25.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:25.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:25.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:25.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:25.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:32:25.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:32:25.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:25.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:25.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f63d8003840 2021-03-09T10:32:25.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:25.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:25.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0069550 con 0x7f63d8003840 2021-03-09T10:32:25.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:32:25.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:25.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:32:25.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:25.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.674+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17175 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc023f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:25.701 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:25.679+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47695 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:25.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.680+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02be20 con 0x7f63d8003840 2021-03-09T10:32:25.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.680+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:25.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.680+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:25.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.680+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:25.703 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:25.703 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:25.704 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:25.704 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:25.704 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:25.704 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:25.705 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:25.705 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:25.705 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:25.705 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:25.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.680+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8003840 2021-03-09T10:32:25.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.680+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:25.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.680+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:32:25.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:25.681+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f4b0 con 0x7f63d8003840 2021-03-09T10:32:26.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:26.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:26.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:26.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:27.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:27.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:27.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:27.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:28.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:28.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:28.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:28.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:29.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:29.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:29.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:29.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:30.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.550+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:30.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.636+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:30.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.636+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:30.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.636+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:30.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:30.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:30.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17175 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc023f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:30.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17175 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc023f00).stop 2021-03-09T10:32:30.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:30.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:30.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:30.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=CLOSED pgs=17175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:30.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:32:30.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:30.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:30.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:30.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:30.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:30.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:30.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:30.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb70 con 0x7f63d8003400 2021-03-09T10:32:30.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:30.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:32:30.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:30.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63e0018390 2021-03-09T10:32:30.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:30.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:30.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:30.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:30.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:30.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.680+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:30.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:32:30.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:30.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:30.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:30.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:30.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:30.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:30.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:30.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:30.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:30.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:30.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:32:30.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:32:30.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:30.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:30.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b260 con 0x7f63e0018390 2021-03-09T10:32:30.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:30.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:30.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018200 con 0x7f63e0018390 2021-03-09T10:32:30.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:32:30.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.681+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:30.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.683+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17190 cs=0 l=1 rev1=1 rx=0x7f63f0032f50 tx=0x7f63f00092b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:30.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.683+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:32:30.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.683+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:30.694 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:30.687+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47733 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:30.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63e0018390 2021-03-09T10:32:30.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.688+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:30.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.688+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:30.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.688+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:30.695 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:30.695 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:30.696 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:30.696 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:30.696 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:30.696 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:30.697 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:30.697 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:30.697 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:30.697 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:30.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.692+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63e0018390 2021-03-09T10:32:30.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.692+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:30.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.692+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:32:30.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:30.692+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63e0018390 2021-03-09T10:32:31.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:31.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:31.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:31.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:32.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:32.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:32.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:32.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:33.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:33.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:33.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:33.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:34.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:34.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:34.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:34.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:35.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.551+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:35.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:35.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:35.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:32:35.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:35.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17190 cs=0 l=1 rev1=1 rx=0x7f63f0032f50 tx=0x7f63f00092b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:35.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17190 cs=0 l=1 rev1=1 rx=0x7f63f0032f50 tx=0x7f63f00092b0).stop 2021-03-09T10:32:35.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:35.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:35.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:35.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:35.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:32:35.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:35.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:35.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:35.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:35.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:35.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:35.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:35.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ff0 con 0x7f63d8003400 2021-03-09T10:32:35.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:35.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011fb0 con 0x7f63d8009130 2021-03-09T10:32:35.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:35.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:32:35.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:35.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:35.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:35.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:35.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:35.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:35.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:32:35.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:35.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.688+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:35.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:35.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:35.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:35.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:35.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:35.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:35.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:35.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:35.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:32:35.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:32:35.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:35.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:35.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013d00 con 0x7f63d8003840 2021-03-09T10:32:35.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:35.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:35.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b260 con 0x7f63d8003840 2021-03-09T10:32:35.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:32:35.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:35.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17201 cs=0 l=1 rev1=1 rx=0x7f63e0065e90 tx=0x7f63f005fee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:35.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:32:35.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.689+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:35.704 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:35.696+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47774 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:35.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63d8003840 2021-03-09T10:32:35.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.696+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:35.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.696+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:35.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.696+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:35.706 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:35.707 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:35.707 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:35.707 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:35.707 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:35.708 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:35.708 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:35.708 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:35.708 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:35.708 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:35.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027250 con 0x7f63d8003840 2021-03-09T10:32:35.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.696+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:35.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.696+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:32:35.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:35.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f740 con 0x7f63d8003840 2021-03-09T10:32:36.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:36.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:36.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:36.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:37.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:37.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:37.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:37.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:38.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:38.637+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:38.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:38.637+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:39.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:39.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:39.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:40.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.552+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:40.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:40.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:40.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.638+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:40.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:40.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:40.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17201 cs=0 l=1 rev1=1 rx=0x7f63e0065e90 tx=0x7f63f005fee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:40.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 secure :-1 s=READY pgs=17201 cs=0 l=1 rev1=1 rx=0x7f63e0065e90 tx=0x7f63f005fee0).stop 2021-03-09T10:32:40.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:40.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:40.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00220d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:40.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00220d0 unknown :-1 s=CLOSED pgs=17201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:40.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:32:40.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:40.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:40.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:40.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:40.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:40.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:40.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:40.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bfc0 con 0x7f63d8003400 2021-03-09T10:32:40.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:40.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bb70 con 0x7f63d8009130 2021-03-09T10:32:40.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:40.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63e0018390 2021-03-09T10:32:40.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.696+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:40.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:40.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:40.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:40.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:40.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:40.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:32:40.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:40.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:40.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:40.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:40.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:40.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:40.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:40.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:40.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:40.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:40.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:32:40.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:32:40.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:40.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:40.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019800 con 0x7f63e0018390 2021-03-09T10:32:40.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:40.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:40.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013d00 con 0x7f63e0018390 2021-03-09T10:32:40.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.698+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:32:40.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.698+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:40.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.698+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17216 cs=0 l=1 rev1=1 rx=0x7f63f004b040 tx=0x7f63f003e400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:40.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.698+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:32:40.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.698+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:40.711 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:40.706+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47812 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:40.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.707+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63e0018390 2021-03-09T10:32:40.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.707+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:40.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.707+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:40.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.707+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:40.712 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:40.713 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:40.713 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:40.713 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:40.713 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:40.713 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:40.714 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:40.714 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:40.714 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:40.714 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:40.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.707+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b6c0 con 0x7f63e0018390 2021-03-09T10:32:40.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.707+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:40.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.707+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:32:40.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:40.708+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f740 con 0x7f63e0018390 2021-03-09T10:32:41.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:41.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:41.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:41.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:42.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:42.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:42.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:42.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:43.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:43.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:43.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:43.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:44.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:44.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:44.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:44.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:45.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.553+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:45.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:45.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:45.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:45.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:45.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17216 cs=0 l=1 rev1=1 rx=0x7f63f004b040 tx=0x7f63f003e400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17216 cs=0 l=1 rev1=1 rx=0x7f63f004b040 tx=0x7f63f003e400).stop 2021-03-09T10:32:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:32:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:45.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:45.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028230 con 0x7f63d8003400 2021-03-09T10:32:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:45.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:32:45.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:45.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003840 2021-03-09T10:32:45.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:45.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:45.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:45.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:45.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.706+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:45.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:45.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:32:45.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:45.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:45.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:45.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:45.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:45.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:45.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:32:45.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:32:45.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:45.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.707+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:45.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.708+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:32:45.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.708+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:45.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.708+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:45.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.708+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f63d8003840 2021-03-09T10:32:45.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.708+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:32:45.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.708+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:45.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.708+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:32:45.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.708+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:45.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.708+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 secure :-1 s=READY pgs=17230 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:45.721 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:45.713+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47853 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:45.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.715+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02be20 con 0x7f63d8003840 2021-03-09T10:32:45.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.715+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:45.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.715+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:45.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.715+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:45.722 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:45.723 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:45.723 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:45.723 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:45.723 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:45.723 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:45.724 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:45.724 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:45.724 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:45.724 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:45.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.719+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02be20 con 0x7f63d8003840 2021-03-09T10:32:45.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.719+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:45.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.719+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:32:45.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:45.719+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:32:46.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:46.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:46.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:46.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:47.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:47.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:47.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:47.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:48.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:48.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:48.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:48.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:49.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:49.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:49.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:49.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:50.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.553+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:50.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:50.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:50.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.638+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:50.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:50.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:50.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 secure :-1 s=READY pgs=17230 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:50.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 secure :-1 s=READY pgs=17230 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc009a00).stop 2021-03-09T10:32:50.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:50.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:50.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:50.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 unknown :-1 s=CLOSED pgs=17230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:50.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:32:50.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:50.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:50.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:50.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:50.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:50.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:50.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:50.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028040 con 0x7f63d8003400 2021-03-09T10:32:50.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:50.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ca30 con 0x7f63d8009130 2021-03-09T10:32:50.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:50.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a7e0 con 0x7f63e0018390 2021-03-09T10:32:50.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:50.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:50.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:50.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:50.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.713+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:50.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:50.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.714+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:32:50.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.714+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:50.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.714+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:50.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:50.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:50.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:50.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:50.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:50.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:50.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:32:50.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:32:50.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:50.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:50.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e2d0 con 0x7f63e0018390 2021-03-09T10:32:50.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:50.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:50.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019800 con 0x7f63e0018390 2021-03-09T10:32:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:32:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.715+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:50.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.716+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:32:50.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.716+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17246 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f005ffc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:50.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.716+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:50.732 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:50.719+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47891 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63e0018390 2021-03-09T10:32:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.720+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.720+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.720+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:50.734 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:50.735 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:50.736 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:50.736 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:50.736 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:50.736 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:50.736 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:50.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.721+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63e0018390 2021-03-09T10:32:50.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.721+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:50.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.721+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:32:50.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:50.722+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63e0018390 2021-03-09T10:32:51.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:51.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:51.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:51.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:52.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:52.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:52.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:52.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:53.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:53.638+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:53.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:53.638+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:54.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:54.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:54.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:54.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:55.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.554+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:32:55.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:55.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:55.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:32:55.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:32:55.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17246 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f005ffc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:32:55.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17246 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f005ffc0).stop 2021-03-09T10:32:55.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:55.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:32:55.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:32:55.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:32:55.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:32:55.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:55.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:32:55.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:55.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:32:55.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:32:55.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:32:55.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:55.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:32:55.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:55.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00689c0 con 0x7f63d8009130 2021-03-09T10:32:55.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:32:55.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00303e0 con 0x7f63d8003840 2021-03-09T10:32:55.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:32:55.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:32:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.720+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:32:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.721+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:32:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.721+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:32:55.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.721+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:32:55.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.721+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:32:55.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.721+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:32:55.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.721+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:32:55.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.721+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:32:55.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.721+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:32:55.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:32:55.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:55.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:55.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:32:55.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:32:55.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:32:55.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:32:55.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:32:55.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:55.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d350 con 0x7f63d8003840 2021-03-09T10:32:55.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:32:55.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:32:55.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:32:55.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:32:55.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.722+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:55.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.723+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:32:55.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.723+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 secure :-1 s=READY pgs=17263 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc012eb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:32:55.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.723+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:32:55.736 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:32:55.726+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47930 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:32:55.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066d40 con 0x7f63d8003840 2021-03-09T10:32:55.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.727+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:32:55.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.727+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:32:55.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.727+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:32:55.737 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:32:55.737 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:32:55.738 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:55.738 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:32:55.738 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:32:55.738 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:32:55.739 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:32:55.739 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:32:55.739 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:32:55.739 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:32:55.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc066d40 con 0x7f63d8003840 2021-03-09T10:32:55.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.727+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:32:55.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.727+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:32:55.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:55.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02bc10 con 0x7f63d8003840 2021-03-09T10:32:56.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:56.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:56.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:56.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:57.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:57.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:57.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:57.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:58.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:58.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:58.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:58.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:32:59.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:59.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:32:59.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:32:59.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:00.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.555+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:00.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.639+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:00.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.639+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:00.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.639+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:00.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:00.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:00.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 secure :-1 s=READY pgs=17263 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc012eb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:00.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 secure :-1 s=READY pgs=17263 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc012eb0).stop 2021-03-09T10:33:00.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:00.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:00.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:00.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 unknown :-1 s=CLOSED pgs=17263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:00.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:33:00.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:00.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000ad0 con 0x7f63d8003400 2021-03-09T10:33:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005df50 con 0x7f63d8009130 2021-03-09T10:33:00.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:00.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c930 con 0x7f63e0018390 2021-03-09T10:33:00.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:00.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:00.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.727+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:33:00.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:00.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:00.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:00.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:33:00.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:33:00.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:00.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:00.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002bb70 con 0x7f63e0018390 2021-03-09T10:33:00.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:00.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:00.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e2d0 con 0x7f63e0018390 2021-03-09T10:33:00.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:33:00.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.728+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:00.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.729+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17271 cs=0 l=1 rev1=1 rx=0x7f63e0048e80 tx=0x7f63f0057f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:00.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.729+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:33:00.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.729+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:00.742 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:00.734+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 47970 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:00.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63e0018390 2021-03-09T10:33:00.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.735+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:00.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.735+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:00.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.735+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:00.743 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:00.744 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:00.744 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:00.744 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:00.744 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:00.744 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:00.745 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:00.745 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:00.745 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:00.745 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:00.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:33:00.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.735+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:00.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.735+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:33:00.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:00.736+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63e0018390 2021-03-09T10:33:01.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:01.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:01.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:01.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:02.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:02.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:02.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:02.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:03.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:03.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:03.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:03.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:04.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:04.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:04.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:04.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:05.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.556+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:05.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:05.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:05.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:05.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:05.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17271 cs=0 l=1 rev1=1 rx=0x7f63e0048e80 tx=0x7f63f0057f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:05.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17271 cs=0 l=1 rev1=1 rx=0x7f63e0048e80 tx=0x7f63f0057f80).stop 2021-03-09T10:33:05.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:05.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:05.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:05.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:05.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:33:05.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:05.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:05.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:05.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:05.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:05.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:05.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:05.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a7e0 con 0x7f63d8003400 2021-03-09T10:33:05.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:05.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:33:05.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:05.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025330 con 0x7f63d8003840 2021-03-09T10:33:05.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:05.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:05.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:05.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:05.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:05.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:05.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:33:05.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:05.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:05.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:05.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:05.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:05.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.736+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:05.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:05.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:05.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:05.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:05.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:33:05.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:33:05.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:05.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:05.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c3e0 con 0x7f63d8003840 2021-03-09T10:33:05.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:05.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:05.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d350 con 0x7f63d8003840 2021-03-09T10:33:05.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:33:05.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:05.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:33:05.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 secure :-1 s=READY pgs=17284 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc071280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:05.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.737+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:05.762 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:05.742+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48009 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:05.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63d8003840 2021-03-09T10:33:05.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.742+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:05.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.742+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:05.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.742+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:05.767 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:05.767 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:05.768 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:05.768 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:05.768 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:05.769 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:05.769 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:05.770 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:05.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc078640 con 0x7f63d8003840 2021-03-09T10:33:05.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.742+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:05.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.742+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:33:05.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:05.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8003840 2021-03-09T10:33:06.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:06.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:06.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:06.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:07.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:07.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:07.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:07.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:08.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:08.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:08.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:08.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:09.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:09.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:09.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:09.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:10.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.557+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:10.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.640+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:10.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.640+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:10.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.641+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:10.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:10.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:10.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 secure :-1 s=READY pgs=17284 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc071280).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 secure :-1 s=READY pgs=17284 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc071280).stop 2021-03-09T10:33:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079de0 unknown :-1 s=CLOSED pgs=17284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:33:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:10.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:10.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021a80 con 0x7f63d8003400 2021-03-09T10:33:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005cfa0 con 0x7f63d8009130 2021-03-09T10:33:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:10.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00303e0 con 0x7f63e0018390 2021-03-09T10:33:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:10.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:33:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.742+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:10.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.743+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:10.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.743+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:10.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.743+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:10.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.743+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:10.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.743+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:10.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.743+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:10.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:10.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:10.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:33:10.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:33:10.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:10.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:10.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d30 con 0x7f63e0018390 2021-03-09T10:33:10.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:10.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:10.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c3e0 con 0x7f63e0018390 2021-03-09T10:33:10.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:33:10.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:10.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17299 cs=0 l=1 rev1=1 rx=0x7f63fc03fee0 tx=0x7f63fc02df70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:10.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:33:10.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.744+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:10.758 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:10.749+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48049 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:10.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066c60 con 0x7f63e0018390 2021-03-09T10:33:10.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.751+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:10.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.751+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:10.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.751+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:10.760 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:10.761 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:10.761 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:10.761 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:10.761 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:10.761 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:10.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63e0018390 2021-03-09T10:33:10.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.751+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:10.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.751+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:33:10.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:10.751+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc030260 con 0x7f63e0018390 2021-03-09T10:33:11.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:11.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:11.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:11.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:12.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:12.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:12.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:13.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:13.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:13.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:13.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:14.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:14.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:14.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:14.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:15.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.558+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:15.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.641+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:15.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.641+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:15.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:15.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:15.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17299 cs=0 l=1 rev1=1 rx=0x7f63fc03fee0 tx=0x7f63fc02df70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:15.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17299 cs=0 l=1 rev1=1 rx=0x7f63fc03fee0 tx=0x7f63fc02df70).stop 2021-03-09T10:33:15.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:15.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:15.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:15.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:15.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:33:15.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:15.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63d8003400 2021-03-09T10:33:15.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:15.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:33:15.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:15.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266f0 con 0x7f63d8003840 2021-03-09T10:33:15.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:15.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:15.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:15.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:15.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:15.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:15.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:33:15.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:15.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.750+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:15.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.751+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:15.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.751+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:15.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.751+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:15.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:15.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:15.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:15.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:15.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:15.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:33:15.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:33:15.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:15.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:15.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:33:15.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:15.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:15.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d30 con 0x7f63d8003840 2021-03-09T10:33:15.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:33:15.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.752+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:15.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.753+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079d90 secure :-1 s=READY pgs=17315 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc050fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:15.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.753+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:33:15.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.753+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:15.765 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:15.757+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48088 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:15.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63d8003840 2021-03-09T10:33:15.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.757+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:15.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.758+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:15.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.758+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:15.767 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:15.767 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:15.767 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:15.767 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:15.767 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:15.768 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:15.768 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.761+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63d8003840 2021-03-09T10:33:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.761+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.761+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:33:15.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:15.762+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc031070 con 0x7f63d8003840 2021-03-09T10:33:16.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:16.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:16.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:16.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:17.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:17.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:17.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:17.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:18.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:18.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:18.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:18.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:19.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:19.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:19.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:19.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:20.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.559+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:20.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:20.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:20.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.642+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:20.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:20.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:20.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079d90 secure :-1 s=READY pgs=17315 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc050fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:20.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079d90 secure :-1 s=READY pgs=17315 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc050fb0).stop 2021-03-09T10:33:20.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:20.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:20.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:20.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079d90 unknown :-1 s=CLOSED pgs=17315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:20.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:33:20.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:20.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:20.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:20.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:20.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:20.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:20.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:20.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025330 con 0x7f63d8003400 2021-03-09T10:33:20.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:20.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000ad0 con 0x7f63d8009130 2021-03-09T10:33:20.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:20.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065af0 con 0x7f63e0018390 2021-03-09T10:33:20.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:20.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:20.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:20.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:20.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.757+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:20.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:20.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.758+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:33:20.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.758+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:20.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.758+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:20.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.758+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:20.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.758+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:20.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.758+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:20.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:20.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:20.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:20.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:20.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:20.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:33:20.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:33:20.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:20.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:20.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63e0018390 2021-03-09T10:33:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002bb70 con 0x7f63e0018390 2021-03-09T10:33:20.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:33:20.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:20.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17326 cs=0 l=1 rev1=1 rx=0x7f63e0029310 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:20.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.760+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:33:20.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.760+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:20.772 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:20.763+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48128 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:20.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63e0018390 2021-03-09T10:33:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.764+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.764+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.764+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:20.773 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:20.774 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:20.774 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:20.774 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:20.774 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:20.774 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:20.775 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:20.775 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:20.775 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:20.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.768+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0041e90 con 0x7f63e0018390 2021-03-09T10:33:20.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.768+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:20.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.768+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041b20 con 0x7f63e0018390 2021-03-09T10:33:20.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:20.768+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:33:21.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:21.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:21.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:21.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:22.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:22.642+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:22.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:22.642+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:23.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:23.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:23.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:23.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:24.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:24.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:24.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:24.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:25.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.560+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:25.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:25.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:25.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:25.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:25.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17326 cs=0 l=1 rev1=1 rx=0x7f63e0029310 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:25.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=17326 cs=0 l=1 rev1=1 rx=0x7f63e0029310 tx=0x7f63f0000bc0).stop 2021-03-09T10:33:25.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:25.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:25.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:25.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=17326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:25.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:33:25.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:25.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:25.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:25.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:25.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:25.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:25.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:25.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00303e0 con 0x7f63d8003400 2021-03-09T10:33:25.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:25.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fb90 con 0x7f63d8009130 2021-03-09T10:33:25.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:25.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a7e0 con 0x7f63d8003840 2021-03-09T10:33:25.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:25.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:25.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:25.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:25.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:25.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.765+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:25.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.765+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:33:25.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.765+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:25.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.765+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:25.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:25.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:25.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:25.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:25.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:25.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:25.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:25.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:25.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:33:25.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:33:25.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:25.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:25.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001dd0 con 0x7f63d8003840 2021-03-09T10:33:25.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:25.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:25.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63d8003840 2021-03-09T10:33:25.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:33:25.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.766+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:25.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.767+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:33:25.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.767+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:25.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.767+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015a90 secure :-1 s=READY pgs=17346 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007ae10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:25.780 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:25.770+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48167 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:25.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038470 con 0x7f63d8003840 2021-03-09T10:33:25.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.771+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:25.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.771+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:25.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.771+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:25.781 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:25.782 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:25.782 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:25.782 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:25.782 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:25.783 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:25.783 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:25.783 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:25.783 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:25.783 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:25.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:33:25.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.776+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:25.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.776+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:33:25.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:25.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63d8003840 2021-03-09T10:33:26.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:26.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:26.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:26.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:27.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:27.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:27.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:27.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:28.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:28.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:28.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:28.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:33:29.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4050000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4050000).stop 2021-03-09T10:33:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:33:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f6405551700 10 monclient: get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:33:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4049000).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:29.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.350+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:33:29.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:29.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:29.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:30.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.561+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.643+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.643+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:30.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.643+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:30.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:30.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:30.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015a90 secure :-1 s=READY pgs=17346 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007ae10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:30.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015a90 secure :-1 s=READY pgs=17346 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007ae10).stop 2021-03-09T10:33:30.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:30.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:30.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:30.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015a90 unknown :-1 s=CLOSED pgs=17346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:30.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:33:30.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:30.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:30.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:30.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:30.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:30.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:30.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:30.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:33:30.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:30.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079c00 con 0x7f63d8009130 2021-03-09T10:33:30.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:30.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001c90 con 0x7f63d8005e00 2021-03-09T10:33:30.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.770+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:30.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:30.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:30.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:30.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:30.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:30.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:33:30.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:30.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:30.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:30.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:30.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.771+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:30.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:30.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:30.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:30.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:30.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:30.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:33:30.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:33:30.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:30.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:30.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b7d0 con 0x7f63d8005e00 2021-03-09T10:33:30.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:30.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:30.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001dd0 con 0x7f63d8005e00 2021-03-09T10:33:30.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:33:30.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.772+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:30.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.773+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=17353 cs=0 l=1 rev1=1 rx=0x7f63f0055860 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:30.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.773+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:33:30.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.773+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:30.787 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:30.777+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48207 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:30.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00556a0 con 0x7f63d8005e00 2021-03-09T10:33:30.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.777+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:30.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.777+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:30.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.777+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:30.788 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:30.789 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:30.789 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:30.789 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:30.789 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:30.789 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:30.790 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:30.790 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:30.790 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:30.790 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:30.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:33:30.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.777+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:30.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.777+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:33:30.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:30.778+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8005e00 2021-03-09T10:33:31.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:31.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:31.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:31.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:32.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:32.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:32.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:32.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:33.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:33.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:33.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:33.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:34.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:34.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:34.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:34.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:35.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.562+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:35.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:35.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:35.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:35.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:35.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=17353 cs=0 l=1 rev1=1 rx=0x7f63f0055860 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:35.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=17353 cs=0 l=1 rev1=1 rx=0x7f63f0055860 tx=0x7f63f0000c30).stop 2021-03-09T10:33:35.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:35.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:35.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:35.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=17353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:35.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:33:35.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:35.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:35.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:35.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:35.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:35.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:35.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:35.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00689c0 con 0x7f63d8003400 2021-03-09T10:33:35.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:35.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000ad0 con 0x7f63d8009130 2021-03-09T10:33:35.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:35.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:33:35.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:35.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:35.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.777+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:35.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.778+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:35.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.778+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:35.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.778+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:35.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.778+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:33:35.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.778+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:35.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.778+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:35.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.779+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:35.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.779+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:35.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.779+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:35.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:35.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:35.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:35.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:35.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:35.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:33:35.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:33:35.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:35.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:35.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c3e0 con 0x7f63d8003840 2021-03-09T10:33:35.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:35.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:35.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b7d0 con 0x7f63d8003840 2021-03-09T10:33:35.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:33:35.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.780+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:35.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.781+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a240 secure :-1 s=READY pgs=17371 cs=0 l=1 rev1=1 rx=0x7f63e005d1c0 tx=0x7f63f0038d90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:35.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.781+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:33:35.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.781+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:35.793 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:35.783+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48246 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:35.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.784+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00286b0 con 0x7f63d8003840 2021-03-09T10:33:35.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.784+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:35.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.784+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:35.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.784+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:35.794 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:35.795 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:35.795 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:35.795 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:35.795 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:35.795 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:35.796 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:35.796 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:35.796 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:35.796 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:35.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028480 con 0x7f63d8003840 2021-03-09T10:33:35.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.787+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:35.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.788+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:33:35.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:35.788+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63d8003840 2021-03-09T10:33:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:36.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:36.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:36.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:37.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:37.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:37.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:37.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:38.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:38.644+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:38.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:38.644+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:39.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:39.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:39.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:39.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:40.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.563+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:40.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:40.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.645+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:40.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:40.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a240 secure :-1 s=READY pgs=17371 cs=0 l=1 rev1=1 rx=0x7f63e005d1c0 tx=0x7f63f0038d90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:40.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a240 secure :-1 s=READY pgs=17371 cs=0 l=1 rev1=1 rx=0x7f63e005d1c0 tx=0x7f63f0038d90).stop 2021-03-09T10:33:40.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:40.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:40.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002a240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:40.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a240 unknown :-1 s=CLOSED pgs=17371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:40.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:33:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:40.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:40.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:40.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:40.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068bc0 con 0x7f63d8003400 2021-03-09T10:33:40.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00303e0 con 0x7f63d8009130 2021-03-09T10:33:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:33:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:40.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:40.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:33:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:40.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.784+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:40.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:40.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:40.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:40.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:40.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:40.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:40.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:40.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:40.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:33:40.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:33:40.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:40.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:40.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce40 con 0x7f63d8005e00 2021-03-09T10:33:40.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:40.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:40.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c3e0 con 0x7f63d8005e00 2021-03-09T10:33:40.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:33:40.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.785+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:40.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=17382 cs=0 l=1 rev1=1 rx=0x7f63f0013470 tx=0x7f63f007b620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:40.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.786+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:33:40.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.786+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:40.799 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:40.790+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48286 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:40.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.790+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007b7f0 con 0x7f63d8005e00 2021-03-09T10:33:40.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.790+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:40.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.790+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:40.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.790+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:40.801 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:40.801 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:40.801 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:40.801 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:40.801 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:40.802 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:40.802 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:40.802 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:40.802 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:40.803 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:40.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.791+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004c260 con 0x7f63d8005e00 2021-03-09T10:33:40.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.791+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:40.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.791+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:33:40.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:40.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:33:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:41.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:41.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:42.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:42.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:42.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:42.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:43.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:43.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:43.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:43.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:44.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:44.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:44.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:45.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.564+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:45.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:45.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:45.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:45.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:45.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=17382 cs=0 l=1 rev1=1 rx=0x7f63f0013470 tx=0x7f63f007b620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:45.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=17382 cs=0 l=1 rev1=1 rx=0x7f63f0013470 tx=0x7f63f007b620).stop 2021-03-09T10:33:45.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:45.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:45.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:45.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=17382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:45.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:33:45.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:45.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:45.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:45.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:45.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:45.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:45.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:45.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00669d0 con 0x7f63d8003400 2021-03-09T10:33:45.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:45.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:33:45.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:45.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00296f0 con 0x7f63d8003840 2021-03-09T10:33:45.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:45.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:45.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:45.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:45.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.791+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:45.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.792+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:45.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.792+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:33:45.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.792+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:45.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.792+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:45.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.792+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:45.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.792+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:45.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.792+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:45.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:45.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:45.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:45.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:45.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:45.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:33:45.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:33:45.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:45.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:45.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00689a0 con 0x7f63d8003840 2021-03-09T10:33:45.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:45.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:45.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce40 con 0x7f63d8003840 2021-03-09T10:33:45.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:33:45.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:45.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:33:45.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 secure :-1 s=READY pgs=17397 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0041dc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:45.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.793+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:45.807 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:45.798+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48325 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:45.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.799+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:33:45.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.799+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:45.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.799+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:45.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.799+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:45.808 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:45.808 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:45.809 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:45.810 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:45.810 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:45.810 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:45.810 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:45.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.799+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:33:45.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.799+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:45.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.799+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:33:45.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:45.800+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:33:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:46.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:46.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:47.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:47.645+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:47.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:47.645+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:48.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:48.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:48.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:48.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:49.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:49.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:49.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:49.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:50.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.565+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:50.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:50.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:50.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.646+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:50.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:33:50.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:50.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 secure :-1 s=READY pgs=17397 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0041dc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:50.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 secure :-1 s=READY pgs=17397 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0041dc0).stop 2021-03-09T10:33:50.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:50.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:50.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:50.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 unknown :-1 s=CLOSED pgs=17397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:50.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:33:50.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:50.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:50.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:50.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:50.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:50.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:50.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:50.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:33:50.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:50.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:33:50.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:50.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b8b0 con 0x7f63d8005e00 2021-03-09T10:33:50.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.798+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:50.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.799+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:50.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.799+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:50.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.799+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:50.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.799+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:50.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.800+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:50.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.800+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:33:50.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.800+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:50.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.800+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:50.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.800+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:50.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.800+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:50.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.800+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:50.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:50.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:50.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:50.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:50.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:50.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:33:50.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:33:50.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:50.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:50.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0069530 con 0x7f63d8005e00 2021-03-09T10:33:50.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:50.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:50.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00689a0 con 0x7f63d8005e00 2021-03-09T10:33:50.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:33:50.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.801+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:50.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.802+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:33:50.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.802+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:50.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.802+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 secure :-1 s=READY pgs=17416 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f0040ea0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:50.815 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:50.805+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48365 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:50.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.806+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8005e00 2021-03-09T10:33:50.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.806+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:50.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.806+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:50.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.806+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:50.817 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:50.817 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:50.817 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:50.817 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:50.817 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:50.818 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:50.818 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:50.818 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:50.818 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:50.818 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:50.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.809+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:33:50.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.809+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:50.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.809+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:33:50.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:50.811+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:33:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:51.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:51.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:52.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:52.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:52.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:52.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:53.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:53.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:53.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:53.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:54.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:54.646+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:54.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:54.646+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:55.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.566+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:33:55.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:55.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:55.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:33:55.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:33:55.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 secure :-1 s=READY pgs=17416 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f0040ea0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:33:55.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 secure :-1 s=READY pgs=17416 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f0040ea0).stop 2021-03-09T10:33:55.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:55.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:33:55.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:33:55.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 unknown :-1 s=CLOSED pgs=17416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:33:55.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:33:55.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:55.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:33:55.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:55.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:33:55.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:33:55.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:33:55.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:55.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066200 con 0x7f63d8003400 2021-03-09T10:33:55.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:55.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00303e0 con 0x7f63d8009130 2021-03-09T10:33:55.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:33:55.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00296f0 con 0x7f63d8003840 2021-03-09T10:33:55.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.805+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:33:55.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.806+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:33:55.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.806+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:55.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.806+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:33:55.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.806+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:33:55.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.806+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:33:55.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.806+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:33:55.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.806+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:33:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.806+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:33:55.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:33:55.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:33:55.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:33:55.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:33:55.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:55.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:55.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:33:55.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:33:55.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:33:55.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:33:55.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:33:55.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:55.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005cfa0 con 0x7f63d8003840 2021-03-09T10:33:55.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:33:55.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:33:55.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0069530 con 0x7f63d8003840 2021-03-09T10:33:55.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:33:55.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.807+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:55.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.809+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 secure :-1 s=READY pgs=17430 cs=0 l=1 rev1=1 rx=0x7f63f000c010 tx=0x7f63f0041f80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:33:55.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.809+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:33:55.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.809+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:33:55.830 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:33:55.813+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48404 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:33:55.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002f5b0 con 0x7f63d8003840 2021-03-09T10:33:55.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.814+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:33:55.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.814+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:33:55.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.814+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:33:55.833 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:33:55.834 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:33:55.834 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:55.835 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:33:55.835 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:33:55.835 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:33:55.836 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:33:55.836 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:33:55.837 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:33:55.837 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:33:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002f3b0 con 0x7f63d8003840 2021-03-09T10:33:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.817+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:33:55.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.817+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:33:55.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:55.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:33:56.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:56.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:56.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:56.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:57.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:57.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:57.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:57.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:58.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:58.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:58.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:58.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:33:59.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:59.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:33:59.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:33:59.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:00.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.567+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:00.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:00.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:00.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.647+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:00.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:00.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 secure :-1 s=READY pgs=17430 cs=0 l=1 rev1=1 rx=0x7f63f000c010 tx=0x7f63f0041f80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 secure :-1 s=READY pgs=17430 cs=0 l=1 rev1=1 rx=0x7f63f000c010 tx=0x7f63f0041f80).stop 2021-03-09T10:34:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:00.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029ff0 unknown :-1 s=CLOSED pgs=17430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:34:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:00.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000ad0 con 0x7f63d8003400 2021-03-09T10:34:00.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b8b0 con 0x7f63d8009130 2021-03-09T10:34:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:34:00.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:00.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:00.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:00.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:00.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:00.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:00.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.814+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:34:00.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.815+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:00.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.815+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:00.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.815+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:00.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.815+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:00.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.815+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:00.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.815+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:00.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.815+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:00.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.815+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:00.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.815+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:34:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:34:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:00.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e060 con 0x7f63d8005e00 2021-03-09T10:34:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018200 con 0x7f63d8005e00 2021-03-09T10:34:00.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:34:00.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.816+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:00.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.817+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 secure :-1 s=READY pgs=17438 cs=0 l=1 rev1=1 rx=0x7f63e006bb70 tx=0x7f63fc075fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:00.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.817+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:34:00.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.817+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:00.953 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:00.821+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48444 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:00.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.822+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc078230 con 0x7f63d8005e00 2021-03-09T10:34:00.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.822+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:00.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.822+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:00.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.822+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:00.955 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:00.955 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:00.955 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:00.956 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:00.956 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:00.956 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:00.956 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:00.957 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.827+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc078230 con 0x7f63d8005e00 2021-03-09T10:34:00.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.827+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.827+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:34:00.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:00.827+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc066dc0 con 0x7f63d8005e00 2021-03-09T10:34:01.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:01.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:01.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:01.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:02.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:02.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:02.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:02.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:03.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:03.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:03.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:03.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:04.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:04.647+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:04.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:04.647+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:05.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.568+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:05.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:05.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:05.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:05.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:05.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 secure :-1 s=READY pgs=17438 cs=0 l=1 rev1=1 rx=0x7f63e006bb70 tx=0x7f63fc075fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:05.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 secure :-1 s=READY pgs=17438 cs=0 l=1 rev1=1 rx=0x7f63e006bb70 tx=0x7f63fc075fe0).stop 2021-03-09T10:34:05.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:05.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:05.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021c80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:05.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021c80 unknown :-1 s=CLOSED pgs=17438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:05.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:34:05.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:05.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:05.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:05.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:05.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:05.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:05.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:05.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00303e0 con 0x7f63d8003400 2021-03-09T10:34:05.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:05.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8009130 2021-03-09T10:34:05.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:05.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:34:05.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:05.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:05.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:05.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:05.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:05.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:05.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:34:05.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:05.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.822+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:05.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.823+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:05.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.823+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:05.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.823+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:05.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:05.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:05.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:05.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:05.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:05.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:34:05.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:34:05.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:05.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:05.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00266a0 con 0x7f63d8003840 2021-03-09T10:34:05.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:05.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:05.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e060 con 0x7f63d8003840 2021-03-09T10:34:05.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:34:05.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.824+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:05.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.825+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e90 secure :-1 s=READY pgs=17455 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc065f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:05.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.825+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:34:05.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.825+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:05.838 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:05.829+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48483 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:05.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.830+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fe80 con 0x7f63d8003840 2021-03-09T10:34:05.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.830+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:05.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.830+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.830+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:05.840 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:05.841 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:05.841 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:05.841 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:05.841 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:05.841 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:05.842 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:05.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fe80 con 0x7f63d8003840 2021-03-09T10:34:05.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.832+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:05.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.832+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:34:05.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:05.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8003840 2021-03-09T10:34:06.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:06.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:06.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:06.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:07.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:07.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:07.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:07.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:08.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:08.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:08.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:08.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:09.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:09.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:09.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:09.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:10.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.569+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:10.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:10.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:10.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.648+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:10.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:10.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:10.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e90 secure :-1 s=READY pgs=17455 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc065f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:10.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e90 secure :-1 s=READY pgs=17455 cs=0 l=1 rev1=1 rx=0x7f63fc001220 tx=0x7f63fc065f20).stop 2021-03-09T10:34:10.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:10.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:10.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029e90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:10.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e90 unknown :-1 s=CLOSED pgs=17455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:10.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:34:10.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:10.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:10.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:10.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:10.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:10.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:10.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:10.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:34:10.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:10.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b8b0 con 0x7f63d8009130 2021-03-09T10:34:10.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:10.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00303e0 con 0x7f63d8005e00 2021-03-09T10:34:10.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:10.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:10.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:34:10.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:10.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:10.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:10.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.830+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:10.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.831+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:10.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.831+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:10.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.831+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:10.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.831+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:10.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.831+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:10.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:10.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:10.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:10.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:10.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:10.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:34:10.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:34:10.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:10.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:10.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b700 con 0x7f63d8005e00 2021-03-09T10:34:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005cfa0 con 0x7f63d8005e00 2021-03-09T10:34:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:34:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.832+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.833+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17470 cs=0 l=1 rev1=1 rx=0x7f63e003a140 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.833+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:34:10.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.833+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:10.847 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:10.838+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48523 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.838+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8005e00 2021-03-09T10:34:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.838+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:10.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.838+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.838+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:10.849 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:10.850 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:10.850 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:10.850 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:10.850 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:10.851 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:34:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.840+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:10.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.840+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:34:10.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:10.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8005e00 2021-03-09T10:34:11.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:11.648+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:11.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:11.648+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:12.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:12.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:12.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:12.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:13.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:13.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:13.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:13.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:14.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:14.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:14.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:14.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:15.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.570+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:15.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:15.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:15.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:15.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:15.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17470 cs=0 l=1 rev1=1 rx=0x7f63e003a140 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:15.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17470 cs=0 l=1 rev1=1 rx=0x7f63e003a140 tx=0x7f63f004db20).stop 2021-03-09T10:34:15.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:15.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:15.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:15.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=17470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:15.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:34:15.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:15.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:15.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:15.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:15.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:15.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:15.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:15.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e320 con 0x7f63d8003400 2021-03-09T10:34:15.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:15.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:34:15.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:15.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003840 2021-03-09T10:34:15.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:15.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:15.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:15.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:15.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.839+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:15.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.840+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:15.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.840+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:34:15.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.840+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:15.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.840+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:15.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.840+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.840+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.840+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:34:15.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:34:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f63d8003840 2021-03-09T10:34:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:15.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b700 con 0x7f63d8003840 2021-03-09T10:34:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:34:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17486 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0066310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:15.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:34:15.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.842+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:15.855 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:15.846+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48562 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:34:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.847+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:15.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.847+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.847+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:15.857 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:15.858 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:15.858 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:15.858 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:15.858 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:15.858 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:15.859 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.848+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:34:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.848+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:15.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.848+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:34:15.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:15.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028d30 con 0x7f63d8003840 2021-03-09T10:34:16.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:16.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:16.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:16.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:17.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:17.649+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:17.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:17.649+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:18.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:18.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:18.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:18.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:19.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:19.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:19.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:19.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:20.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.571+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:20.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:20.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:20.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.650+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:20.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17486 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0066310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17486 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0066310).stop 2021-03-09T10:34:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:20.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:20.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=17486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:20.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:34:20.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:20.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c5d0 con 0x7f63d8003400 2021-03-09T10:34:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b8b0 con 0x7f63d8009130 2021-03-09T10:34:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:20.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b510 con 0x7f63d8005e00 2021-03-09T10:34:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:20.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.847+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.848+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:20.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.848+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.848+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:34:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.848+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.848+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:20.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.848+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.848+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.848+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:20.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:34:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:34:20.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b700 con 0x7f63d8005e00 2021-03-09T10:34:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:20.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f63d8005e00 2021-03-09T10:34:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:34:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:34:20.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:20.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17493 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:20.862 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:20.853+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48602 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.853+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027e50 con 0x7f63d8005e00 2021-03-09T10:34:20.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.853+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.853+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:20.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.853+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:20.865 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:20.866 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:20.866 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:20.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8005e00 2021-03-09T10:34:20.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.854+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:20.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.854+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:34:20.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:20.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:34:21.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:21.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:21.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:21.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:22.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:22.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:22.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:22.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:23.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:23.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:23.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:23.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:24.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:24.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:24.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:24.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:25.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.572+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:25.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:25.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:25.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17493 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17493 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006c30).stop 2021-03-09T10:34:25.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:25.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=17493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:34:25.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:25.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:25.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:25.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:25.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:34:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:25.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63d8009130 2021-03-09T10:34:25.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:25.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b310 con 0x7f63d8003840 2021-03-09T10:34:25.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:25.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:25.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:25.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:25.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.854+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:25.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.855+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.855+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:34:25.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.855+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.855+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:25.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:25.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:25.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:25.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:25.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:25.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:25.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:25.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:34:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:34:25.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:25.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:25.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8003840 2021-03-09T10:34:25.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:25.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:25.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b700 con 0x7f63d8003840 2021-03-09T10:34:25.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:34:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.856+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.857+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:34:25.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.857+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17512 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0031da0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:25.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.857+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:25.882 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:25.862+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48641 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007a610 con 0x7f63d8003840 2021-03-09T10:34:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.862+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.862+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:25.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.862+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:25.884 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:25.885 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:25.886 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:25.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.863+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a610 con 0x7f63d8003840 2021-03-09T10:34:25.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.863+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:25.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.863+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:34:25.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:25.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:34:26.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:26.650+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:26.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:26.650+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:27.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:27.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:27.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:27.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:28.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:28.651+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:28.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:28.651+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:29.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:29.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:29.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:29.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:30.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.573+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:30.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:30.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:30.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.652+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:30.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17512 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0031da0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17512 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0031da0).stop 2021-03-09T10:34:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:30.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=17512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:34:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:30.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:30.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00303e0 con 0x7f63d8003400 2021-03-09T10:34:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:34:30.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b8b0 con 0x7f63d8005e00 2021-03-09T10:34:30.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:30.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.862+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.863+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:30.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.863+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.863+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.863+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:34:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.863+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.863+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:30.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.863+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.863+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.863+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:30.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:34:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:34:30.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027c60 con 0x7f63d8005e00 2021-03-09T10:34:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:30.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T10:34:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:34:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.864+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.865+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17524 cs=0 l=1 rev1=1 rx=0x7f63f003e320 tx=0x7f63f002d300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:30.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.865+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:34:30.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.865+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:30.877 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:30.870+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48681 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:30.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00112f0 con 0x7f63d8005e00 2021-03-09T10:34:30.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.871+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:30.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.871+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:30.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.871+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:30.879 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:30.879 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:30.879 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:30.879 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:30.880 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:30.881 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00112f0 con 0x7f63d8005e00 2021-03-09T10:34:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.873+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:30.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.873+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:34:30.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:30.874+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0057c10 con 0x7f63d8005e00 2021-03-09T10:34:31.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:31.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:31.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:31.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:32.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:32.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:32.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:32.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:33.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:33.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:33.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:33.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:34.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:34.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:34.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:34.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:35.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.574+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:35.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:35.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:35.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17524 cs=0 l=1 rev1=1 rx=0x7f63f003e320 tx=0x7f63f002d300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17524 cs=0 l=1 rev1=1 rx=0x7f63f003e320 tx=0x7f63f002d300).stop 2021-03-09T10:34:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:35.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=17524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:34:35.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:35.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:35.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008ef0 con 0x7f63d8003400 2021-03-09T10:34:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00147c0 con 0x7f63d8009130 2021-03-09T10:34:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067330 con 0x7f63d8003840 2021-03-09T10:34:35.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:35.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:34:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:35.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.871+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:35.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:35.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:35.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:34:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:34:35.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.873+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b700 con 0x7f63d8003840 2021-03-09T10:34:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:35.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027c60 con 0x7f63d8003840 2021-03-09T10:34:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:34:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17542 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:35.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:34:35.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.874+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:35.887 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:35.878+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48720 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:34:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.879+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:35.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.879+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.879+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:35.889 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:35.890 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:35.891 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63d8003840 2021-03-09T10:34:35.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.883+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.883+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:34:35.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:35.884+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001dbb0 con 0x7f63d8003840 2021-03-09T10:34:36.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:36.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:36.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:36.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:37.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:37.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:37.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:37.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:38.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:38.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:38.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:38.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:39.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:39.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:39.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:39.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:40.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.575+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:40.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:40.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.653+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:40.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:40.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:40.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17542 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17542 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f000df50).stop 2021-03-09T10:34:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:40.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=17542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:34:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:40.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:40.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ad80 con 0x7f63d8003400 2021-03-09T10:34:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b310 con 0x7f63d8009130 2021-03-09T10:34:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:40.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:34:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:40.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.879+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.880+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:34:40.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.880+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:40.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.880+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:40.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:40.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:40.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.880+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:40.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:40.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:40.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:40.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:34:40.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:34:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d750 con 0x7f63d8005e00 2021-03-09T10:34:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:40.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b700 con 0x7f63d8005e00 2021-03-09T10:34:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:34:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:40.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17549 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f003e1f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:34:40.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.881+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:40.895 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:40.885+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48760 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:34:40.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.886+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.886+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.886+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:40.897 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:40.898 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:40.898 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:40.898 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:40.898 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:40.899 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.887+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:34:40.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.887+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.887+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:34:40.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:40.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028d30 con 0x7f63d8005e00 2021-03-09T10:34:41.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:41.652+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:41.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:41.652+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:42.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:42.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:42.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:42.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:43.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:43.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:43.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:43.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:44.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:44.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:44.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:44.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:45.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.576+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:45.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:45.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:45.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:34:45.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17549 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f003e1f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=17549 cs=0 l=1 rev1=1 rx=0x7f63f002d370 tx=0x7f63f003e1f0).stop 2021-03-09T10:34:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:45.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:45.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:45.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=17549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:45.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:34:45.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:45.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068470 con 0x7f63d8003400 2021-03-09T10:34:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008ef0 con 0x7f63d8009130 2021-03-09T10:34:45.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:45.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011a30 con 0x7f63d8003840 2021-03-09T10:34:45.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:45.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:45.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.886+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:45.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.887+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.887+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.887+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.887+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:34:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.887+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:45.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.887+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.888+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.888+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.888+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:45.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:45.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:34:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:34:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:45.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b700 con 0x7f63d8003840 2021-03-09T10:34:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d750 con 0x7f63d8003840 2021-03-09T10:34:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:34:45.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:34:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17567 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0023de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:45.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.889+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:45.901 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:45.893+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48799 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:34:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.894+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.894+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:45.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.894+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:45.903 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:45.904 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:45.904 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:45.904 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:45.904 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:45.904 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:34:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.897+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.897+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:34:45.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:45.899+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028d30 con 0x7f63d8003840 2021-03-09T10:34:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:46.653+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:46.653+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:47.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:47.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:47.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:47.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:48.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:48.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:48.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:48.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:49.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:49.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:49.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:49.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:50.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.577+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:50.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:50.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:50.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.654+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:50.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17567 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0023de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17567 cs=0 l=1 rev1=1 rx=0x7f63f0014010 tx=0x7f63f0023de0).stop 2021-03-09T10:34:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:50.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=17567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:50.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:34:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:50.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:50.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015410 con 0x7f63d8003400 2021-03-09T10:34:50.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:50.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8009130 2021-03-09T10:34:50.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:50.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b8b0 con 0x7f63d8005e00 2021-03-09T10:34:50.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:50.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.894+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:50.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.895+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.895+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:34:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.895+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.895+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.895+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:50.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.895+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.895+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:50.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:34:50.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:34:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002fe60 con 0x7f63d8005e00 2021-03-09T10:34:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:50.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00266a0 con 0x7f63d8005e00 2021-03-09T10:34:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:34:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.896+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.897+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=17578 cs=0 l=1 rev1=1 rx=0x7f63e00678c0 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:50.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.897+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:34:50.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.897+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:50.911 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:50.902+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48839 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:50.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f63d8005e00 2021-03-09T10:34:50.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.904+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:50.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.904+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:50.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.904+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:50.912 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:50.913 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:50.914 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:50.914 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:50.914 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:50.914 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:50.915 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f63d8005e00 2021-03-09T10:34:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.907+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.907+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:34:50.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:50.908+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:34:51.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:51.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:51.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:51.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:52.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:52.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:52.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:52.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:53.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:53.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:53.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:53.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:54.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:54.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:54.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:54.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:55.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.578+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:34:55.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:55.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:55.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:34:55.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:34:55.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=17578 cs=0 l=1 rev1=1 rx=0x7f63e00678c0 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:34:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 secure :-1 s=READY pgs=17578 cs=0 l=1 rev1=1 rx=0x7f63e00678c0 tx=0x7f63fc009990).stop 2021-03-09T10:34:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:55.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:34:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:34:55.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=17578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:34:55.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:34:55.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:55.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:34:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:34:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:34:55.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:34:55.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:55.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b310 con 0x7f63d8003400 2021-03-09T10:34:55.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:55.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:34:55.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:34:55.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00125f0 con 0x7f63d8003840 2021-03-09T10:34:55.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:34:55.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:34:55.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:55.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:34:55.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.903+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:34:55.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.904+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:34:55.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.904+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:34:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.904+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:34:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.904+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:34:55.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.904+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:34:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.904+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:34:55.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.904+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:34:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:34:55.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:34:55.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:34:55.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:34:55.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:34:55.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:34:55.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a3c0 con 0x7f63d8003840 2021-03-09T10:34:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:34:55.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:34:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b700 con 0x7f63d8003840 2021-03-09T10:34:55.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:34:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.905+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17595 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:34:55.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.906+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:34:55.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.906+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:34:55.930 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:34:55.912+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48878 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:34:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.912+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:34:55.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.912+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:34:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.912+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:34:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.912+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:34:55.933 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:34:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:34:55.934 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:55.935 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:34:55.937 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:34:55.937 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:34:55.938 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:34:55.938 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:34:55.938 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:34:55.939 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:34:55.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:34:55.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.913+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:34:55.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.913+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:34:55.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:55.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:34:56.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:56.654+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:56.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:56.654+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:57.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:57.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:57.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:57.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:58.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:58.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:58.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:58.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:34:59.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:59.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:34:59.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:34:59.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:00.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.579+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:00.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:00.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:00.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.656+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:00.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:00.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:00.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17595 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:00.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=17595 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0006c30).stop 2021-03-09T10:35:00.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:00.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:00.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:00.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=17595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:00.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:35:00.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:00.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:00.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b8b0 con 0x7f63d8003400 2021-03-09T10:35:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8009130 2021-03-09T10:35:00.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:35:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:00.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:00.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:35:00.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.913+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.914+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.914+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:00.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.914+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:00.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:00.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:00.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:00.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:00.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:00.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:35:00.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:35:00.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:00.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:00.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b700 con 0x7f63d8005e00 2021-03-09T10:35:00.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:00.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:00.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002fe60 con 0x7f63d8005e00 2021-03-09T10:35:00.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:35:00.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.915+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:00.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023a70 secure :-1 s=READY pgs=17608 cs=0 l=1 rev1=1 rx=0x7f63e007aa30 tx=0x7f63fc012320).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:00.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.916+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:35:00.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.916+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:00.928 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:00.920+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48918 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:00.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:35:00.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.920+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:00.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.920+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:00.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.920+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:00.930 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:00.930 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:00.930 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:00.930 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:00.931 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:00.932 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8005e00 2021-03-09T10:35:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.924+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:00.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.924+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:35:00.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:00.924+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:35:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:01.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:01.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:01.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:02.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:02.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:02.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:02.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:03.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:03.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:03.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:03.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:04.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:04.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:04.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:04.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:05.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.580+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:05.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:05.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:05.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0023a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:05.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0023a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:05.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023a70 secure :-1 s=READY pgs=17608 cs=0 l=1 rev1=1 rx=0x7f63e007aa30 tx=0x7f63fc012320).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:05.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023a70 secure :-1 s=READY pgs=17608 cs=0 l=1 rev1=1 rx=0x7f63e007aa30 tx=0x7f63fc012320).stop 2021-03-09T10:35:05.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:05.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0023a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0023a70 unknown :-1 s=CLOSED pgs=17608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:35:05.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:05.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:05.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:05.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b310 con 0x7f63d8003400 2021-03-09T10:35:05.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002efa0 con 0x7f63d8009130 2021-03-09T10:35:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b9c0 con 0x7f63d8003840 2021-03-09T10:35:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:05.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:35:05.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.921+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:05.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:05.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:05.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:05.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:05.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:05.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:05.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:05.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002bf40 con 0x7f63d8003840 2021-03-09T10:35:05.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:05.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:05.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a3c0 con 0x7f63d8003840 2021-03-09T10:35:05.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.923+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:05.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.923+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:05.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.923+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:35:05.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.923+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 secure :-1 s=READY pgs=17620 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:05.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.923+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:05.940 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:05.928+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48957 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:05.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:35:05.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.928+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:05.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.928+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:05.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.928+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:05.942 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:05.942 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:05.942 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:05.942 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:05.943 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:05.943 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:05.943 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:05.943 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:05.944 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:05.944 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:05.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007f650 con 0x7f63d8003840 2021-03-09T10:35:05.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.933+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:05.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.933+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:35:05.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:05.934+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:35:06.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:06.655+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:06.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:06.655+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:07.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:07.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:08.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:08.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:08.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:08.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:09.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:09.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:10.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:09.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:10.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.581+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:10.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:10.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:10.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.657+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:10.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 secure :-1 s=READY pgs=17620 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 secure :-1 s=READY pgs=17620 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004de00).stop 2021-03-09T10:35:10.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:10.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=17620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:35:10.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:10.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:10.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:10.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:10.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:35:10.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:10.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015650 con 0x7f63d8009130 2021-03-09T10:35:10.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:10.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:35:10.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:10.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:10.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:10.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:10.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:10.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:10.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:35:10.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:10.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.929+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:10.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.930+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:10.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.930+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:10.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.930+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:10.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:10.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:10.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:10.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:10.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016af0 con 0x7f63d8005e00 2021-03-09T10:35:10.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:10.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:10.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002bf40 con 0x7f63d8005e00 2021-03-09T10:35:10.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:10.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.931+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:10.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.932+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079460 secure :-1 s=READY pgs=17638 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f007f6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.932+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:35:10.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.932+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:10.954 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:10.936+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 48997 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:10.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8005e00 2021-03-09T10:35:10.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.937+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:10.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.937+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.937+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:10.957 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:10.958 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:10.959 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:10.960 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:10.960 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:10.961 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:10.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8005e00 2021-03-09T10:35:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.940+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:10.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8005e00 2021-03-09T10:35:10.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:10.940+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:35:11.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:11.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:11.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:11.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:12.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:12.656+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:12.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:12.656+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:13.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:13.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:13.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:13.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:14.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:14.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:14.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:14.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:15.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.582+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:15.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:15.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:15.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:15.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:15.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079460 secure :-1 s=READY pgs=17638 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f007f6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:15.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079460 secure :-1 s=READY pgs=17638 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f007f6c0).stop 2021-03-09T10:35:15.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:15.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:15.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:15.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079460 unknown :-1 s=CLOSED pgs=17638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:15.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:35:15.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:15.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:15.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:15.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:15.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:15.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:15.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:15.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T10:35:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a6c0 con 0x7f63d8009130 2021-03-09T10:35:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b310 con 0x7f63d8003840 2021-03-09T10:35:15.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.937+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.938+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.938+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.938+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:15.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.938+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.938+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.938+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:35:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.938+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:15.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.938+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:15.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:15.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:15.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002fe60 con 0x7f63d8003840 2021-03-09T10:35:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016af0 con 0x7f63d8003840 2021-03-09T10:35:15.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:15.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.939+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:15.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.940+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:35:15.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.940+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:15.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.940+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017c10 secure :-1 s=READY pgs=17646 cs=0 l=1 rev1=1 rx=0x7f63f0058070 tx=0x7f63f001d410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:15.952 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:15.944+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49036 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:15.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00112f0 con 0x7f63d8003840 2021-03-09T10:35:15.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.944+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:15.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.944+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:15.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.944+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:15.954 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:15.954 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:15.954 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:15.954 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:15.955 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:15.955 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:15.955 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:15.955 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:15.956 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:15.956 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:15.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8003840 2021-03-09T10:35:15.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.945+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:15.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.945+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:35:15.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:15.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028d30 con 0x7f63d8003840 2021-03-09T10:35:16.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:16.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:16.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:16.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:17.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:17.657+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:17.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:17.657+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:18.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:18.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:18.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:18.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:19.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:19.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:19.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:19.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:20.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.583+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:20.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:20.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:20.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.658+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:20.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:20.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:20.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017c10 secure :-1 s=READY pgs=17646 cs=0 l=1 rev1=1 rx=0x7f63f0058070 tx=0x7f63f001d410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:20.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017c10 secure :-1 s=READY pgs=17646 cs=0 l=1 rev1=1 rx=0x7f63f0058070 tx=0x7f63f001d410).stop 2021-03-09T10:35:20.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:20.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:20.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0017c10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:20.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0017c10 unknown :-1 s=CLOSED pgs=17646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:20.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:35:20.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:20.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:20.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.944+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:20.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068630 con 0x7f63d8003400 2021-03-09T10:35:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002efa0 con 0x7f63d8009130 2021-03-09T10:35:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:35:20.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:20.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:20.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:20.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:20.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.945+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:20.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.946+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:20.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.946+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:35:20.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.946+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:20.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.946+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:20.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:20.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:20.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:20.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:20.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:20.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:20.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:20.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:20.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:20.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:20.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:20.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f0f0 con 0x7f63d8005e00 2021-03-09T10:35:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:20.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002fe60 con 0x7f63d8005e00 2021-03-09T10:35:20.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:20.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:20.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.947+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 secure :-1 s=READY pgs=17658 cs=0 l=1 rev1=1 rx=0x7f63f0035fd0 tx=0x7f63f007ac20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:20.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.948+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:35:20.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.948+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:20.959 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:20.951+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49076 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:20.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.952+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8005e00 2021-03-09T10:35:20.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.952+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.952+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.952+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:20.961 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:20.962 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:20.963 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8005e00 2021-03-09T10:35:20.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.953+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.953+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:35:20.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:20.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f004c260 con 0x7f63d8005e00 2021-03-09T10:35:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:21.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:21.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:21.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:22.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:22.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:22.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:22.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:23.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:23.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:23.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:23.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:24.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:24.658+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:24.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:24.658+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:25.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.584+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:25.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:25.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:25.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:35:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 secure :-1 s=READY pgs=17658 cs=0 l=1 rev1=1 rx=0x7f63f0035fd0 tx=0x7f63f007ac20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 secure :-1 s=READY pgs=17658 cs=0 l=1 rev1=1 rx=0x7f63f0035fd0 tx=0x7f63f007ac20).stop 2021-03-09T10:35:25.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:25.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:25.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:25.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 unknown :-1 s=CLOSED pgs=17658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:25.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:35:25.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:25.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:25.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:25.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.951+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:25.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019d10 con 0x7f63d8003400 2021-03-09T10:35:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:25.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011c30 con 0x7f63d8009130 2021-03-09T10:35:25.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:25.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:35:25.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:25.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:25.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:25.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:25.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:25.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:25.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:35:25.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:25.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:25.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:25.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:25.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.952+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:25.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:25.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:25.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:25.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:25.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:25.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:25.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:25.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c710 con 0x7f63d8003840 2021-03-09T10:35:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:25.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:25.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f0f0 con 0x7f63d8003840 2021-03-09T10:35:25.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:25.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:25.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:35:25.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba60 secure :-1 s=READY pgs=17671 cs=0 l=1 rev1=1 rx=0x7f63f0006d40 tx=0x7f63f0032570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:25.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.953+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:25.966 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:25.960+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49115 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:25.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:35:25.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.960+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:25.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.960+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:25.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.960+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:25.968 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:25.968 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:25.969 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:25.970 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:25.970 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:25.970 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:25.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0031c80 con 0x7f63d8003840 2021-03-09T10:35:25.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.960+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:25.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.960+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:35:25.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:25.961+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8003840 2021-03-09T10:35:26.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:26.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:26.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:26.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:27.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:27.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:27.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:27.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:28.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:28.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:28.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:28.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:29.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:29.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:29.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:29.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:30.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.585+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:30.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:30.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:30.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.659+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:30.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:30.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba60 secure :-1 s=READY pgs=17671 cs=0 l=1 rev1=1 rx=0x7f63f0006d40 tx=0x7f63f0032570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba60 secure :-1 s=READY pgs=17671 cs=0 l=1 rev1=1 rx=0x7f63f0006d40 tx=0x7f63f0032570).stop 2021-03-09T10:35:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:30.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002ba60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba60 unknown :-1 s=CLOSED pgs=17671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:35:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:30.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:30.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067520 con 0x7f63d8003400 2021-03-09T10:35:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8009130 2021-03-09T10:35:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:30.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cbd0 con 0x7f63d8005e00 2021-03-09T10:35:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:30.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:30.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.961+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:30.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.961+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:35:30.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.961+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:30.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.961+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:30.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.961+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:30.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.961+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:30.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.961+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:30.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:30.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:30.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:30.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:30.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:30.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:35:30.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:35:30.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:30.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:30.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002fe60 con 0x7f63d8005e00 2021-03-09T10:35:30.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:30.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b700 con 0x7f63d8005e00 2021-03-09T10:35:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:35:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.962+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.963+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 secure :-1 s=READY pgs=17685 cs=0 l=1 rev1=1 rx=0x7f63e00168c0 tx=0x7f63fc02bba0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:30.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.963+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:35:30.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.963+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:30.974 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:30.967+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49155 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:30.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02bdd0 con 0x7f63d8005e00 2021-03-09T10:35:30.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.968+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:30.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.968+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:30.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.968+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:30.976 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:30.977 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:30.978 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:30.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.973+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02bdd0 con 0x7f63d8005e00 2021-03-09T10:35:30.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.973+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:30.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.973+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:35:30.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:30.974+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:35:31.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:31.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:31.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:31.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:32.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:32.659+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:32.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:32.659+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:33.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:33.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:33.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:33.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:34.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:34.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:34.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:34.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:35.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.586+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:35.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:35.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:35.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 secure :-1 s=READY pgs=17685 cs=0 l=1 rev1=1 rx=0x7f63e00168c0 tx=0x7f63fc02bba0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:35.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 secure :-1 s=READY pgs=17685 cs=0 l=1 rev1=1 rx=0x7f63e00168c0 tx=0x7f63fc02bba0).stop 2021-03-09T10:35:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:35.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ad0 unknown :-1 s=CLOSED pgs=17685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:35.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:35:35.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:35.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:35.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:35.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:35.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:35.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:35.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:35.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T10:35:35.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:35.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a730 con 0x7f63d8009130 2021-03-09T10:35:35.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:35.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011a30 con 0x7f63d8003840 2021-03-09T10:35:35.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:35.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:35.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:35.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:35.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:35.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.969+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:35.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.969+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:35:35.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.969+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:35.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.969+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:35.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.969+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:35.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.969+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:35.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.969+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:35.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:35.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:35.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:35.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:35.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:35.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:35.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:35.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:35.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:35.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b2c0 con 0x7f63d8003840 2021-03-09T10:35:35.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:35.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:35.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c710 con 0x7f63d8003840 2021-03-09T10:35:35.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.970+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.971+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:35:35.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 secure :-1 s=READY pgs=17699 cs=0 l=1 rev1=1 rx=0x7f63f003e340 tx=0x7f63f0006c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:35.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.971+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:35.983 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:35.975+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49194 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:35.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:35:35.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.976+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.976+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.976+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:35.985 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:35.986 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:35.986 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:35.986 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:35.986 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:35.987 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:35.987 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:35.987 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:35.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:35:35.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.978+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:35.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.978+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:35:35.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:35.981+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:35:36.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:36.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:37.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:36.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:37.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:37.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:37.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:37.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:38.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:38.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:38.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:38.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:39.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:39.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:39.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:39.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:40.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.586+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:40.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:40.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:40.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.660+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:40.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:40.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 secure :-1 s=READY pgs=17699 cs=0 l=1 rev1=1 rx=0x7f63f003e340 tx=0x7f63f0006c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 secure :-1 s=READY pgs=17699 cs=0 l=1 rev1=1 rx=0x7f63f003e340 tx=0x7f63f0006c30).stop 2021-03-09T10:35:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:40.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006d510 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006d510 unknown :-1 s=CLOSED pgs=17699 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:35:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:40.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:40.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63d8003400 2021-03-09T10:35:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ac20 con 0x7f63d8009130 2021-03-09T10:35:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:40.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:35:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:40.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:35:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:40.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:40.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.975+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:40.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:40.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:40.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:40.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e006d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:40.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:40.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:40.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:40.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:40.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:40.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:40.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f63d8005e00 2021-03-09T10:35:40.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:40.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:40.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b2c0 con 0x7f63d8005e00 2021-03-09T10:35:40.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:40.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.976+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:40.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.977+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007b120 secure :-1 s=READY pgs=17712 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0055290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:40.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.977+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:35:40.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.977+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:40.991 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:40.981+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49234 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:40.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007ac90 con 0x7f63d8005e00 2021-03-09T10:35:40.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.984+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:40.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.984+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:40.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.984+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:40.993 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:40.993 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:40.994 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:40.994 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:40.994 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:40.994 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:40.994 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:40.995 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.986+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007ac90 con 0x7f63d8005e00 2021-03-09T10:35:40.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.986+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:40.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.986+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:35:40.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:40.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0060c50 con 0x7f63d8005e00 2021-03-09T10:35:41.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:41.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:41.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:41.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:42.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:42.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:42.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:42.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:43.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:43.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:43.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:43.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:44.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:44.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:44.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:44.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:45.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.587+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:45.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:45.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:45.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:45.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:45.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007b120 secure :-1 s=READY pgs=17712 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0055290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:45.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007b120 secure :-1 s=READY pgs=17712 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0055290).stop 2021-03-09T10:35:45.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:45.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:45.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e007b120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:45.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e007b120 unknown :-1 s=CLOSED pgs=17712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:35:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:45.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:45.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:45.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T10:35:45.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:45.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cbd0 con 0x7f63d8009130 2021-03-09T10:35:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000b9c0 con 0x7f63d8003840 2021-03-09T10:35:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:45.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:45.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:35:45.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:45.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.982+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:45.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:45.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:45.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:45.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:45.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:45.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:45.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:45.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:45.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:35:45.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:35:45.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:45.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:45.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00694e0 con 0x7f63d8003840 2021-03-09T10:35:45.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:45.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:45.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002fe60 con 0x7f63d8003840 2021-03-09T10:35:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:35:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.983+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079fc0 secure :-1 s=READY pgs=17725 cs=0 l=1 rev1=1 rx=0x7f63fc0583a0 tx=0x7f63fc0583d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.984+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:35:45.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.984+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:46.000 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:45.988+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49273 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02bc80 con 0x7f63d8003840 2021-03-09T10:35:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.989+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:46.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.989+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.989+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:46.002 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:46.003 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:46.004 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:35:46.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.989+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:46.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.989+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:35:46.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:45.991+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058000 con 0x7f63d8003840 2021-03-09T10:35:46.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:46.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:46.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:46.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:47.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:47.660+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:47.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:47.660+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:48.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:48.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:48.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:48.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:49.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:49.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:49.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:49.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:50.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.588+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:50.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:50.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:50.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.661+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:50.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:35:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:51.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079fc0 secure :-1 s=READY pgs=17725 cs=0 l=1 rev1=1 rx=0x7f63fc0583a0 tx=0x7f63fc0583d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:51.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079fc0 secure :-1 s=READY pgs=17725 cs=0 l=1 rev1=1 rx=0x7f63fc0583a0 tx=0x7f63fc0583d0).stop 2021-03-09T10:35:51.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:51.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:51.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079fc0 unknown :-1 s=CLOSED pgs=17725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:35:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:51.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:51.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:51.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:51.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8003400 2021-03-09T10:35:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b3f0 con 0x7f63d8009130 2021-03-09T10:35:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017430 con 0x7f63d8005e00 2021-03-09T10:35:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:51.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.989+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:51.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:51.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:35:51.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:51.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:51.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:51.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:51.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:51.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:51.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:51.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:51.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:51.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:51.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:51.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:51.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:51.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:51.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e200 con 0x7f63d8005e00 2021-03-09T10:35:51.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:51.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:51.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f63d8005e00 2021-03-09T10:35:51.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:51.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.990+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:51.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.991+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021aa0 secure :-1 s=READY pgs=17739 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0039fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:51.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.991+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:35:51.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.991+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:51.389 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:50.996+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49313 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038ea0 con 0x7f63d8005e00 2021-03-09T10:35:51.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.997+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:51.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.997+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:51.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:50.997+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:51.398 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:51.398 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:51.398 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:51.399 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:51.400 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:51.400 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:51.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007f650 con 0x7f63d8005e00 2021-03-09T10:35:51.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.001+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:51.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.001+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:35:51.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.002+0000 7f6405d52700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T10:35:51.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.002+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:51.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.002+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:51.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.002+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63d8005e00 2021-03-09T10:35:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.002+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021aa0 secure :-1 s=READY pgs=17739 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0039fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.002+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021aa0 secure :-1 s=READY pgs=17739 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0039fb0).stop 2021-03-09T10:35:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:51.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021aa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021aa0 unknown :-1 s=CLOSED pgs=17739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:35:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:51.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:51.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:51.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:35:51.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:51.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63d8009130 2021-03-09T10:35:51.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:51.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017430 con 0x7f63d8003840 2021-03-09T10:35:51.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:51.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:51.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:35:51.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.003+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:51.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:51.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:51.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:51.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:51.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:51.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:51.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:51.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:51.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:51.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f63d8003840 2021-03-09T10:35:51.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:51.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:51.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e200 con 0x7f63d8003840 2021-03-09T10:35:51.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:51.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 secure :-1 s=READY pgs=17751 cs=0 l=1 rev1=1 rx=0x7f63f003a040 tx=0x7f63f005d350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:35:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:51.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:35:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:51.413 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:51.414 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:51.415 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:51.415 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:51.415 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:51.415 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:51.415 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:51.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:35:51.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:51.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.004+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:35:51.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.005+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:35:51.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:51.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:51.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:52.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:52.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:52.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:52.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:53.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:53.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:53.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:53.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:54.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:54.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:54.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:54.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:55.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.589+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:35:55.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.661+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:55.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.661+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:55.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:35:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:35:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 secure :-1 s=READY pgs=17751 cs=0 l=1 rev1=1 rx=0x7f63f003a040 tx=0x7f63f005d350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:35:56.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 secure :-1 s=READY pgs=17751 cs=0 l=1 rev1=1 rx=0x7f63f003a040 tx=0x7f63f005d350).stop 2021-03-09T10:35:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:35:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:35:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 unknown :-1 s=CLOSED pgs=17751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:35:56.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:35:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:35:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:56.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:35:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:35:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:35:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068630 con 0x7f63d8003400 2021-03-09T10:35:56.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029440 con 0x7f63d8009130 2021-03-09T10:35:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:35:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8005e00 2021-03-09T10:35:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:35:56.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:35:56.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:56.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:35:56.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:35:56.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:35:56.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:35:56.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:35:56.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.997+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:35:56.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.999+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:35:56.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.999+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:35:56.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:55.999+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:35:56.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:35:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:35:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:35:56.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:35:56.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:35:56.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:35:56.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:56.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c6e0 con 0x7f63d8005e00 2021-03-09T10:35:56.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:35:56.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:35:56.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f63d8005e00 2021-03-09T10:35:56.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:35:56.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:56.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000af40 secure :-1 s=READY pgs=17763 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f00361e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:35:56.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:35:56.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.000+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:35:56.012 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:35:56.003+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49352 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:35:56.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:35:56.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.003+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:35:56.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.003+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:35:56.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.003+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:35:56.014 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:35:56.015 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:35:56.015 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:56.015 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:35:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:35:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:35:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:35:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:35:56.016 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:35:56.017 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:35:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:35:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.006+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:35:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.006+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:35:56.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.007+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:35:56.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:56.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:56.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:57.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:57.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:57.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:57.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:58.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:58.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:58.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:58.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:35:59.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:59.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:35:59.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:35:59.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:00.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:00.590+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:00.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:00.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:00.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:00.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:00.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:00.662+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:01.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:01.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000af40 secure :-1 s=READY pgs=17763 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f00361e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000af40 secure :-1 s=READY pgs=17763 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f00361e0).stop 2021-03-09T10:36:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:01.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:01.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000af40 unknown :-1 s=CLOSED pgs=17763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:01.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:36:01.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:01.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:01.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:01.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:01.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:01.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:01.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:01.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bfc0 con 0x7f63d8003400 2021-03-09T10:36:01.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:01.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00693a0 con 0x7f63d8009130 2021-03-09T10:36:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024c90 con 0x7f63d8003840 2021-03-09T10:36:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:01.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.004+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.004+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:01.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.004+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.004+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:01.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.004+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:36:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.004+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.004+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.005+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:01.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.005+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.005+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e0079ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:01.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e0079ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:01.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:01.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:01.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065870 con 0x7f63d8003840 2021-03-09T10:36:01.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:01.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c6e0 con 0x7f63d8003840 2021-03-09T10:36:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:01.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:36:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:01.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.006+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 secure :-1 s=READY pgs=17772 cs=0 l=1 rev1=1 rx=0x7f63f007e820 tx=0x7f63f0013430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:01.019 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:01.010+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49392 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:01.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005d480 con 0x7f63d8003840 2021-03-09T10:36:01.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.010+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:01.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.010+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:01.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.010+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:01.022 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:01.022 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:01.022 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:01.022 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:01.023 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:01.023 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:01.023 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:01.023 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:01.023 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:01.024 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:01.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.011+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:36:01.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.011+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:01.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.011+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:36:01.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.013+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:36:01.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:01.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:01.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:02.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:02.662+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:02.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:02.662+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:03.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:03.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:03.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:03.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:04.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:04.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:04.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:04.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:05.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:05.591+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:05.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:05.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:05.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:05.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:06.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 secure :-1 s=READY pgs=17772 cs=0 l=1 rev1=1 rx=0x7f63f007e820 tx=0x7f63f0013430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:06.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 secure :-1 s=READY pgs=17772 cs=0 l=1 rev1=1 rx=0x7f63f007e820 tx=0x7f63f0013430).stop 2021-03-09T10:36:06.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:06.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ad0 unknown :-1 s=CLOSED pgs=17772 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:36:06.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:06.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:06.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:06.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:06.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:36:06.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:06.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63d8009130 2021-03-09T10:36:06.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:06.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ee20 con 0x7f63d8005e00 2021-03-09T10:36:06.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.010+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:06.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:06.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:36:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:06.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.011+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:06.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:06.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00679f0 con 0x7f63d8005e00 2021-03-09T10:36:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:06.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065870 con 0x7f63d8005e00 2021-03-09T10:36:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:06.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.012+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.013+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:36:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.013+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:06.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.013+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17784 cs=0 l=1 rev1=1 rx=0x7f63f003a040 tx=0x7f63f0028fa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:06.026 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:06.017+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49431 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:36:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.017+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.017+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:06.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.017+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:06.028 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:06.028 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:06.028 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:06.028 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:06.028 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:06.029 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:06.029 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:06.029 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:06.029 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:06.029 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8005e00 2021-03-09T10:36:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.023+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.023+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:36:06.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.024+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005dd90 con 0x7f63d8005e00 2021-03-09T10:36:06.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:06.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:06.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:07.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:07.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:07.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:07.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:08.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:08.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:08.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:08.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:09.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:09.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:09.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:09.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:10.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:10.592+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:10.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:10.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:10.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:10.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:10.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:10.663+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:36:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:11.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17784 cs=0 l=1 rev1=1 rx=0x7f63f003a040 tx=0x7f63f0028fa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17784 cs=0 l=1 rev1=1 rx=0x7f63f003a040 tx=0x7f63f0028fa0).stop 2021-03-09T10:36:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:11.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=CLOSED pgs=17784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:36:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00664e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:11.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:11.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00693a0 con 0x7f63d8003400 2021-03-09T10:36:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068630 con 0x7f63d8009130 2021-03-09T10:36:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:11.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017430 con 0x7f63d8003840 2021-03-09T10:36:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.017+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.018+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:11.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.018+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:11.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.018+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:11.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.018+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:36:11.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.018+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:11.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.018+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:11.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:11.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:11.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:11.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:11.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:11.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:11.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c3a0 con 0x7f63d8003840 2021-03-09T10:36:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00679f0 con 0x7f63d8003840 2021-03-09T10:36:11.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:11.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.019+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:11.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.020+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00664e0 secure :-1 s=READY pgs=17799 cs=0 l=1 rev1=1 rx=0x7f63f0036eb0 tx=0x7f63f005dfa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:11.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.020+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:36:11.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.020+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:11.033 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:11.024+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49471 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:11.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0073e10 con 0x7f63d8003840 2021-03-09T10:36:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.025+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.025+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.025+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:11.034 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:11.035 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:11.036 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:11.036 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:11.036 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:11.036 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:36:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.029+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.029+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:36:11.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.029+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8003840 2021-03-09T10:36:11.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.663+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:11.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:11.663+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:12.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:12.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:12.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:12.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:13.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:13.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:13.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:13.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:14.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:14.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:14.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:14.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:15.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:15.593+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:15.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:15.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:15.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:15.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:16.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00664e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:16.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00664e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:16.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00664e0 secure :-1 s=READY pgs=17799 cs=0 l=1 rev1=1 rx=0x7f63f0036eb0 tx=0x7f63f005dfa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:16.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00664e0 secure :-1 s=READY pgs=17799 cs=0 l=1 rev1=1 rx=0x7f63f0036eb0 tx=0x7f63f005dfa0).stop 2021-03-09T10:36:16.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:16.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:16.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00664e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:16.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00664e0 unknown :-1 s=CLOSED pgs=17799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:16.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:36:16.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:16.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:16.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:16.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00191c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:16.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:36:16.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:16.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63d8009130 2021-03-09T10:36:16.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:16.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021550 con 0x7f63d8005e00 2021-03-09T10:36:16.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:16.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:16.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:16.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:16.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:16.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.026+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00191c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:16.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.026+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:36:16.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.026+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:16.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.026+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:16.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:16.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:16.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:16.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:16.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:16.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:16.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:16.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:16.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:16.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00016e0 con 0x7f63d8005e00 2021-03-09T10:36:16.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:16.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:16.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c3a0 con 0x7f63d8005e00 2021-03-09T10:36:16.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:16.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:16.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.027+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00191c0 secure :-1 s=READY pgs=17813 cs=0 l=1 rev1=1 rx=0x7f63f0060fe0 tx=0x7f63f0089030).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:16.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.028+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:36:16.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.028+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:16.046 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:16.031+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49510 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:16.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:36:16.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.032+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:16.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.032+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:16.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.032+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:16.049 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:16.049 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:16.049 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:16.050 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:16.050 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:16.050 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:16.051 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:16.051 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:16.051 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:16.052 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:16.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8005e00 2021-03-09T10:36:16.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.036+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:16.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.036+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:36:16.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.036+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:36:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:16.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:16.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:17.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:17.664+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:17.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:17.664+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:18.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:18.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:18.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:18.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:19.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:19.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:19.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:19.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:20.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:20.594+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:20.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:20.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:20.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:20.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:20.665+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:21.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00191c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:21.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00191c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:21.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00191c0 secure :-1 s=READY pgs=17813 cs=0 l=1 rev1=1 rx=0x7f63f0060fe0 tx=0x7f63f0089030).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:21.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00191c0 secure :-1 s=READY pgs=17813 cs=0 l=1 rev1=1 rx=0x7f63f0060fe0 tx=0x7f63f0089030).stop 2021-03-09T10:36:21.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:21.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:21.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00191c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:21.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00191c0 unknown :-1 s=CLOSED pgs=17813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:21.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:36:21.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:21.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008f10 con 0x7f63d8003400 2021-03-09T10:36:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00015a0 con 0x7f63d8009130 2021-03-09T10:36:21.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065eb0 con 0x7f63d8003840 2021-03-09T10:36:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:21.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.032+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:36:21.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:21.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:21.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:21.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:21.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:21.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:21.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:21.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:21.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:21.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:21.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:36:21.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:36:21.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:21.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:21.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b560 con 0x7f63d8003840 2021-03-09T10:36:21.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:21.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:21.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00694e0 con 0x7f63d8003840 2021-03-09T10:36:21.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:36:21.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.033+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:21.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.034+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c890 secure :-1 s=READY pgs=17822 cs=0 l=1 rev1=1 rx=0x7f63e00090e0 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:21.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.034+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:36:21.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.034+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:21.047 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:21.040+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49550 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:21.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:36:21.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.040+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:21.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.040+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:21.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.040+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:21.048 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:21.049 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:21.049 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:21.049 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:21.049 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:21.049 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:21.050 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:21.050 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:21.050 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:21.051 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:21.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058000 con 0x7f63d8003840 2021-03-09T10:36:21.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.040+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:21.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.040+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:36:21.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:36:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:21.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:21.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:22.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:22.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:22.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:22.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:23.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:23.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:23.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:23.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:24.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:24.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:24.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:25.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:25.595+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:25.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:25.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:25.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:25.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:26.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:26.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:26.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c890 secure :-1 s=READY pgs=17822 cs=0 l=1 rev1=1 rx=0x7f63e00090e0 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:26.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c890 secure :-1 s=READY pgs=17822 cs=0 l=1 rev1=1 rx=0x7f63e00090e0 tx=0x7f63fc04a6d0).stop 2021-03-09T10:36:26.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:26.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:26.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:26.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c890 unknown :-1 s=CLOSED pgs=17822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:26.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:36:26.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:26.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:26.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:26.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:26.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:26.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:26.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:26.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:36:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00693a0 con 0x7f63d8009130 2021-03-09T10:36:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068630 con 0x7f63d8005e00 2021-03-09T10:36:26.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.040+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.041+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:26.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.041+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:26.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.041+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.041+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:36:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.041+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.041+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:26.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.041+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.041+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.041+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:26.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:26.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8005e00 2021-03-09T10:36:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:26.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00016e0 con 0x7f63d8005e00 2021-03-09T10:36:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17843 cs=0 l=1 rev1=1 rx=0x7f63f0073de0 tx=0x7f63f007f6c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:26.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.044+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:36:26.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.044+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:26.055 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:26.047+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49589 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:26.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8005e00 2021-03-09T10:36:26.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.048+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:26.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.048+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.048+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:26.057 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:26.058 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:26.059 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:26.059 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:26.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:36:26.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.050+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:26.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.050+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:36:26.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8005e00 2021-03-09T10:36:26.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.665+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:26.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:26.665+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:27.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:27.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:27.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:27.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:28.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:28.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:28.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:28.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:29.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:29.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:29.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:29.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:30.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:30.596+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:30.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:30.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:30.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:30.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:30.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:30.666+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:31.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.047+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:31.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.047+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.047+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17843 cs=0 l=1 rev1=1 rx=0x7f63f0073de0 tx=0x7f63f007f6c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.047+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17843 cs=0 l=1 rev1=1 rx=0x7f63f0073de0 tx=0x7f63f007f6c0).stop 2021-03-09T10:36:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.047+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.047+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:31.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.047+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:31.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=CLOSED pgs=17843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:31.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:36:31.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:31.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:31.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:31.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63d8003400 2021-03-09T10:36:31.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:31.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8009130 2021-03-09T10:36:31.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:31.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018b50 con 0x7f63d8003840 2021-03-09T10:36:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:31.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:31.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.048+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:31.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.049+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:36:31.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.049+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:31.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.049+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:31.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.049+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:31.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.049+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.049+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:31.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:31.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c3a0 con 0x7f63d8003840 2021-03-09T10:36:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:36:31.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:31.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:31.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=17855 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0060000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:31.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:36:31.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.050+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:31.063 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:31.055+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49628 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:31.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8003840 2021-03-09T10:36:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.057+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.057+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.057+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:31.064 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:31.065 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:31.065 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:31.065 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:31.065 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:31.066 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:31.066 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:31.066 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:31.066 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:31.067 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:31.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8003840 2021-03-09T10:36:31.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.059+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:31.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.059+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:36:31.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:36:31.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:31.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:31.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:32.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:32.666+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:32.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:32.666+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:33.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:33.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:33.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:33.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:34.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:34.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:34.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:34.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:35.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:35.597+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:35.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:35.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:35.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:35.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:36.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:36.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:36.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=17855 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0060000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:36.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=17855 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0060000).stop 2021-03-09T10:36:36.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=CLOSED pgs=17855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:36:36.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:36.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:36.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:36.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:36.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:36.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:36.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:36.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068630 con 0x7f63d8003400 2021-03-09T10:36:36.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:36.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:36:36.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:36.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63d8005e00 2021-03-09T10:36:36.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:36.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:36.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:36.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:36.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.056+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:36.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.057+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:36.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.057+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:36:36.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.057+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:36.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.057+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:36.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:36.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:36.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:36.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:36.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:36.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:36.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:36.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:36.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b7c0 con 0x7f63d8005e00 2021-03-09T10:36:36.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:36.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c3a0 con 0x7f63d8005e00 2021-03-09T10:36:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.058+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17869 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:36.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.059+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:36:36.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.059+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:36.071 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:36.063+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49670 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0088df0 con 0x7f63d8005e00 2021-03-09T10:36:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.064+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.064+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:36.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.064+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:36.073 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:36.074 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:36.074 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:36.074 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:36.074 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:36.075 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:36.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:36:36.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.070+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:36.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.070+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:36:36.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.070+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:36:36.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:36.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:36.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:37.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:37.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:37.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:37.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:38.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:38.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:38.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:38.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:39.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:39.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:39.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:39.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:40.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:40.598+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:40.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:40.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:40.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:40.667+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:41.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17869 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:41.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17869 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f004db20).stop 2021-03-09T10:36:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:41.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=CLOSED pgs=17869 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:41.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:36:41.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.063+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:41.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:41.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:41.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:41.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:41.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:41.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:41.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266f0 con 0x7f63d8003400 2021-03-09T10:36:41.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:41.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00292a0 con 0x7f63d8009130 2021-03-09T10:36:41.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:41.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8003840 2021-03-09T10:36:41.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:41.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:41.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:41.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:41.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:41.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:36:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:41.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:41.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.064+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:41.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:41.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:41.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:41.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:41.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:41.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:41.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c3a0 con 0x7f63d8003840 2021-03-09T10:36:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:41.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:41.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b7c0 con 0x7f63d8003840 2021-03-09T10:36:41.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:41.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:41.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:36:41.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:41.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.066+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 secure :-1 s=READY pgs=17884 cs=0 l=1 rev1=1 rx=0x7f63f007d020 tx=0x7f63f00272c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:41.079 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:41.071+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49708 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:41.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:36:41.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.072+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:41.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.072+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:41.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.072+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:41.081 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:41.081 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:41.081 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:41.081 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:41.081 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:41.082 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:41.082 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:41.082 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:41.082 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:41.082 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:41.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8003840 2021-03-09T10:36:41.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.076+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:41.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.076+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:36:41.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.077+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:36:41.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:41.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:41.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:42.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:42.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:42.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:42.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:43.667+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:43.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:43.667+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:44.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:44.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:44.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:44.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:45.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:45.599+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:45.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:45.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:45.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:45.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:46.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:46.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 secure :-1 s=READY pgs=17884 cs=0 l=1 rev1=1 rx=0x7f63f007d020 tx=0x7f63f00272c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 secure :-1 s=READY pgs=17884 cs=0 l=1 rev1=1 rx=0x7f63f007d020 tx=0x7f63f00272c0).stop 2021-03-09T10:36:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:46.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c7f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 unknown :-1 s=CLOSED pgs=17884 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:36:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:46.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:46.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065eb0 con 0x7f63d8003400 2021-03-09T10:36:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8009130 2021-03-09T10:36:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:46.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023630 con 0x7f63d8005e00 2021-03-09T10:36:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:46.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.072+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:36:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:46.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:46.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:46.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c570 con 0x7f63d8005e00 2021-03-09T10:36:46.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c3a0 con 0x7f63d8005e00 2021-03-09T10:36:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:46.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.073+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:46.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17894 cs=0 l=1 rev1=1 rx=0x7f63f0060070 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:46.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.074+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:36:46.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.074+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:46.087 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:46.078+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49749 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:46.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8005e00 2021-03-09T10:36:46.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.079+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:46.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.079+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:46.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.079+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:46.088 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:46.089 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:46.089 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:46.089 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:46.089 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:46.089 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:46.090 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:46.090 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:46.090 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:46.090 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:46.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:36:46.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.079+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:46.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.079+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:36:46.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:36:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:46.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:46.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:47.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:47.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:47.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:47.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:48.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:48.668+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:48.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:48.668+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:49.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:49.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:49.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:49.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:50.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:50.600+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:50.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:50.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:50.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:50.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:50.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:50.669+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:51.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:36:51.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17894 cs=0 l=1 rev1=1 rx=0x7f63f0060070 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17894 cs=0 l=1 rev1=1 rx=0x7f63f0060070 tx=0x7f63f0000bc0).stop 2021-03-09T10:36:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:51.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=CLOSED pgs=17894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:36:51.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:51.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:51.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:51.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:51.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:51.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:51.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:51.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:36:51.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:51.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d9f0 con 0x7f63d8009130 2021-03-09T10:36:51.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:51.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065eb0 con 0x7f63d8003840 2021-03-09T10:36:51.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:51.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:51.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:36:51.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:51.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.079+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:51.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.080+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:51.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.080+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:51.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.080+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:51.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:51.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:51.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:51.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:51.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:51.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022330 con 0x7f63d8003840 2021-03-09T10:36:51.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:51.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:51.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c570 con 0x7f63d8003840 2021-03-09T10:36:51.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:51.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:51.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 secure :-1 s=READY pgs=17910 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f0055470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:51.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:36:51.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.081+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:51.094 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:51.086+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49787 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:51.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:36:51.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.087+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:51.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.087+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:51.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.087+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:51.096 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:51.096 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:51.096 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:51.096 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:51.097 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:51.097 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:51.097 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:51.097 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:51.097 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:51.098 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:51.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:36:51.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.087+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:51.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.087+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:36:51.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8003840 2021-03-09T10:36:51.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:51.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:51.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:52.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:52.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:52.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:53.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:53.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:53.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:53.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:54.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:54.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:54.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:54.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:55.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:55.601+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:36:55.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:55.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:55.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:55.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:56.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:36:56.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:36:56.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 secure :-1 s=READY pgs=17910 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f0055470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:36:56.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 secure :-1 s=READY pgs=17910 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f0055470).stop 2021-03-09T10:36:56.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:56.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:36:56.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c7f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:36:56.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c7f0 unknown :-1 s=CLOSED pgs=17910 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:36:56.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:36:56.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:36:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:36:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:36:56.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:36:56.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:56.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00094c0 con 0x7f63d8003400 2021-03-09T10:36:56.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:56.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8009130 2021-03-09T10:36:56.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:36:56.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8005e00 2021-03-09T10:36:56.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:36:56.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:36:56.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:36:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:36:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.087+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:36:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:36:56.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:36:56.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:36:56.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:36:56.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:36:56.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:36:56.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:36:56.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:56.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:56.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:36:56.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:36:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:36:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:36:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:36:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:56.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:36:56.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:36:56.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:36:56.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022330 con 0x7f63d8005e00 2021-03-09T10:36:56.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:36:56.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:56.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17924 cs=0 l=1 rev1=1 rx=0x7f63f005df00 tx=0x7f63f0032570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:36:56.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:36:56.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.088+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:36:56.102 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:36:56.094+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49828 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:36:56.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8005e00 2021-03-09T10:36:56.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.094+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:36:56.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.094+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:36:56.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.094+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:36:56.104 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:36:56.104 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:36:56.104 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:56.104 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:36:56.104 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:36:56.105 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:36:56.105 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:36:56.105 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:36:56.105 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:36:56.105 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:36:56.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8005e00 2021-03-09T10:36:56.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.094+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:36:56.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.094+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:36:56.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:36:56.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.669+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:56.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:56.669+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:57.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:57.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:57.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:57.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:58.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:58.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:58.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:58.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:36:59.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:59.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:36:59.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:36:59.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:00.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:00.602+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:00.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:00.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:00.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:00.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:00.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:00.670+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:01.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:01.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:01.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17924 cs=0 l=1 rev1=1 rx=0x7f63f005df00 tx=0x7f63f0032570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:01.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 secure :-1 s=READY pgs=17924 cs=0 l=1 rev1=1 rx=0x7f63f005df00 tx=0x7f63f0032570).stop 2021-03-09T10:37:01.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:01.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:01.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079ab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079ab0 unknown :-1 s=CLOSED pgs=17924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:37:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:01.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:01.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.094+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:01.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:01.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:01.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023100 con 0x7f63d8003400 2021-03-09T10:37:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068630 con 0x7f63d8009130 2021-03-09T10:37:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:01.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:37:01.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:01.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:01.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:01.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:01.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:01.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.096+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:01.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.096+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:37:01.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.096+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:01.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.096+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:01.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.096+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:01.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.096+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:01.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.096+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:01.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:01.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:01.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:01.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:01.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:01.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:01.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:01.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:01.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:01.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0000ab0 con 0x7f63d8003840 2021-03-09T10:37:01.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:01.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:01.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:37:01.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:01.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.097+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:01.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.098+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=17944 cs=0 l=1 rev1=1 rx=0x7f63f0031650 tx=0x7f63f00814d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:01.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.098+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:37:01.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.098+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:01.110 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:01.101+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49868 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:01.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:37:01.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.102+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:01.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.102+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:01.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.102+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:01.111 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:01.112 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:01.112 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:01.112 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:01.112 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:01.113 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:01.113 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:01.113 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:01.113 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:01.113 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:01.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:37:01.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.104+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:01.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.104+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:37:01.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.105+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8003840 2021-03-09T10:37:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.670+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:01.670+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:02.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:02.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:02.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:03.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:03.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:03.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:03.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:04.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:04.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:04.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:04.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:05.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:05.603+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:05.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:05.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:05.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:05.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:06.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:06.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=17944 cs=0 l=1 rev1=1 rx=0x7f63f0031650 tx=0x7f63f00814d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=17944 cs=0 l=1 rev1=1 rx=0x7f63f0031650 tx=0x7f63f00814d0).stop 2021-03-09T10:37:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:06.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=CLOSED pgs=17944 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:37:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:06.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:06.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:06.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:06.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:06.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a730 con 0x7f63d8003400 2021-03-09T10:37:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bfc0 con 0x7f63d8009130 2021-03-09T10:37:06.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018b50 con 0x7f63d8005e00 2021-03-09T10:37:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:06.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.102+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.103+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.103+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:06.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.103+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.103+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:37:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.103+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:06.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.103+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.103+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.103+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.103+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:06.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:06.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:06.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:06.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:06.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:06.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c3a0 con 0x7f63d8005e00 2021-03-09T10:37:06.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0000ab0 con 0x7f63d8005e00 2021-03-09T10:37:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:06.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:37:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=17953 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0057400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:06.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.104+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:06.117 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:06.109+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49909 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8005e00 2021-03-09T10:37:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.110+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:06.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.110+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.110+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:06.119 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:06.120 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:06.121 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:37:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.112+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8005e00 2021-03-09T10:37:06.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.112+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:37:06.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:06.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:06.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:07.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:07.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:07.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:07.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:08.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:08.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:08.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:08.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:09.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:09.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:09.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:09.671+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:10.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:10.604+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:10.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:10.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:10.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:10.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:10.672+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:11.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:11.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=17953 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0057400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=17953 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0057400).stop 2021-03-09T10:37:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:11.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:11.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:11.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=17953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:11.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:37:11.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:11.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:11.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:11.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:37:11.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:11.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b5f0 con 0x7f63d8009130 2021-03-09T10:37:11.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:11.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8003840 2021-03-09T10:37:11.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:11.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:11.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:11.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:11.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:11.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:11.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.111+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:37:11.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.111+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:11.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.111+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:11.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.111+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.111+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.111+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:11.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:11.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:11.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:11.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:11.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:11.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:37:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:11.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c3a0 con 0x7f63d8003840 2021-03-09T10:37:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=17967 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f007e330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:11.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:37:11.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.112+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:11.125 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:11.117+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49947 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:37:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.118+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:11.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.118+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:11.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.118+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:11.127 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:11.127 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:11.127 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:11.127 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:11.128 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:11.128 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:11.128 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:11.129 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:11.129 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:11.129 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:11.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:37:11.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.118+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:11.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.118+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:37:11.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8003840 2021-03-09T10:37:11.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:11.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:11.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:12.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:12.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:12.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:13.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:13.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:13.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:13.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:14.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:14.671+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:14.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:14.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:15.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:15.605+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:15.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:15.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:15.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:15.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:16.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=17967 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f007e330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=17967 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f007e330).stop 2021-03-09T10:37:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:16.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=CLOSED pgs=17967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:37:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:16.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:16.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:16.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:16.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:16.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:16.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:37:16.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:16.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8009130 2021-03-09T10:37:16.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:16.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017fd0 con 0x7f63d8005e00 2021-03-09T10:37:16.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:16.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:16.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:16.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:16.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:16.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:16.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:37:16.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:16.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.118+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:16.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:16.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:16.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:16.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:16.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:16.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:16.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:16.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:16.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:16.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:16.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:16.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:16.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018110 con 0x7f63d8005e00 2021-03-09T10:37:16.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:16.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:16.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8005e00 2021-03-09T10:37:16.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:16.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.119+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:16.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.121+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:37:16.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.121+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=17983 cs=0 l=1 rev1=1 rx=0x7f63f0086d60 tx=0x7f63f0081410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:16.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.121+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:16.133 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:16.125+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 49990 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:16.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f003e180 con 0x7f63d8005e00 2021-03-09T10:37:16.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.126+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:16.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.126+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:16.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.126+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:16.135 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:16.135 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:16.136 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:16.136 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:16.136 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:16.136 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:16.136 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:16.137 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:16.137 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:16.137 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:16.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:37:16.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.128+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:16.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.128+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:37:16.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.128+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038880 con 0x7f63d8005e00 2021-03-09T10:37:16.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.672+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:16.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:16.672+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:17.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:17.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:17.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:17.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:18.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:18.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:18.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:18.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:19.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:19.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:19.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:19.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:20.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:20.606+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:20.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:20.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:20.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:20.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:20.673+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:21.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:21.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:21.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=17983 cs=0 l=1 rev1=1 rx=0x7f63f0086d60 tx=0x7f63f0081410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:21.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=17983 cs=0 l=1 rev1=1 rx=0x7f63f0086d60 tx=0x7f63f0081410).stop 2021-03-09T10:37:21.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:21.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:21.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:21.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=17983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:21.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:37:21.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:21.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:21.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:21.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:21.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:21.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:21.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:21.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018b50 con 0x7f63d8003400 2021-03-09T10:37:21.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:21.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006be30 con 0x7f63d8009130 2021-03-09T10:37:21.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:21.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003840 2021-03-09T10:37:21.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:21.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:21.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:21.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:21.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.126+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:21.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:21.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:37:21.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:21.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:21.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:21.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:21.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:21.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:21.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:21.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:21.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:21.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:21.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002c3a0 con 0x7f63d8003840 2021-03-09T10:37:21.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:21.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:21.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018110 con 0x7f63d8003840 2021-03-09T10:37:21.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:21.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.127+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:21.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.128+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=17993 cs=0 l=1 rev1=1 rx=0x7f63f007d040 tx=0x7f63f00893b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:21.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.128+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:37:21.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.128+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:21.142 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:21.133+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50030 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:21.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:37:21.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.134+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:21.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.134+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:21.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.134+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:21.143 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:21.143 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:21.144 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:21.144 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:21.144 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:21.144 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:21.145 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:21.145 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:21.145 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:21.145 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:21.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:37:21.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.138+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:21.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.138+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:37:21.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0031b90 con 0x7f63d8003840 2021-03-09T10:37:21.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:21.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:21.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:22.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:22.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:22.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:22.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:23.673+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:23.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:23.673+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:24.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:24.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:24.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:24.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:25.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:25.607+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:25.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:25.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:25.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:26.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:26.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:26.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=17993 cs=0 l=1 rev1=1 rx=0x7f63f007d040 tx=0x7f63f00893b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:26.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=17993 cs=0 l=1 rev1=1 rx=0x7f63f007d040 tx=0x7f63f00893b0).stop 2021-03-09T10:37:26.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:26.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:26.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:26.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=CLOSED pgs=17993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:26.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:37:26.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:26.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:26.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:26.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:26.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:26.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:26.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:26.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8003400 2021-03-09T10:37:26.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:26.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b5f0 con 0x7f63d8009130 2021-03-09T10:37:26.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:26.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:37:26.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:26.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:26.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.134+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:26.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.135+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:26.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.135+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:26.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.135+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:26.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.135+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:37:26.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.135+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:26.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.135+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:26.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.135+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:26.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.135+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:26.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.135+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:26.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:26.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:26.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:26.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:26.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:26.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00225d0 con 0x7f63d8005e00 2021-03-09T10:37:26.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:26.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:26.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002c3a0 con 0x7f63d8005e00 2021-03-09T10:37:26.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:26.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.136+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:26.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.137+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18011 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f006c3c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:26.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.137+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:37:26.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.137+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:26.149 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:26.140+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50071 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:26.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:37:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.141+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.141+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.141+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:26.150 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:26.151 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:26.151 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:26.151 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:26.151 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:26.151 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:26.152 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:26.152 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:26.152 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:26.152 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:26.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:37:26.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.141+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:26.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.141+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:37:26.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.143+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0031b90 con 0x7f63d8005e00 2021-03-09T10:37:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:26.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:26.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:27.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:27.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:27.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:27.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:28.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:28.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:28.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:28.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:29.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:29.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:29.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:29.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:30.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:30.608+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:30.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:30.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:30.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:30.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:30.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:30.674+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:31.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:31.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:31.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18011 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f006c3c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:31.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18011 cs=0 l=1 rev1=1 rx=0x7f63f0009330 tx=0x7f63f006c3c0).stop 2021-03-09T10:37:31.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:31.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:31.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:31.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18011 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:31.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:37:31.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:31.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:31.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:31.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:31.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:31.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:31.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:31.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065eb0 con 0x7f63d8003400 2021-03-09T10:37:31.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:31.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8009130 2021-03-09T10:37:31.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:31.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b5f0 con 0x7f63d8003840 2021-03-09T10:37:31.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.141+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:31.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.142+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:31.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.142+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:31.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.142+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:31.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.142+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:31.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.142+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:31.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.142+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:37:31.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.142+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:31.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.142+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:31.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:31.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:31.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:31.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:31.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:31.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:31.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:31.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:31.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:31.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:31.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:31.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:31.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:37:31.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:31.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:31.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00225d0 con 0x7f63d8003840 2021-03-09T10:37:31.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:31.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.144+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:31.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.145+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:37:31.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.145+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 secure :-1 s=READY pgs=18030 cs=0 l=1 rev1=1 rx=0x7f63f005eab0 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:31.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.145+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:31.156 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:31.147+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50109 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:31.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.148+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0032570 con 0x7f63d8003840 2021-03-09T10:37:31.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.148+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:31.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.148+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:31.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.148+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:31.158 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:31.158 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:31.158 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:31.159 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:31.159 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:31.159 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:31.159 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:31.160 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:31.160 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:31.160 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:31.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:37:31.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.150+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:31.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.150+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:37:31.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.151+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0032360 con 0x7f63d8003840 2021-03-09T10:37:31.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:31.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:31.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:32.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:32.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:32.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:33.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:33.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:33.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:33.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:34.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:34.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:34.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:34.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:35.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:35.609+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:35.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:35.674+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:35.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:35.674+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:36.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:36.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:36.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 secure :-1 s=READY pgs=18030 cs=0 l=1 rev1=1 rx=0x7f63f005eab0 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:36.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 secure :-1 s=READY pgs=18030 cs=0 l=1 rev1=1 rx=0x7f63f005eab0 tx=0x7f63f0000c30).stop 2021-03-09T10:37:36.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:36.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:36.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:36.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 unknown :-1 s=CLOSED pgs=18030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:36.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:37:36.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:36.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:36.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:36.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e0b0 con 0x7f63d8003400 2021-03-09T10:37:36.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:36.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00693e0 con 0x7f63d8009130 2021-03-09T10:37:36.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:36.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018b50 con 0x7f63d8005e00 2021-03-09T10:37:36.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.148+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:36.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.149+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:36.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.149+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:36.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.149+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:36.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.149+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:36.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.149+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:36.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.149+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:37:36.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.149+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:36.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.149+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:36.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.150+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:36.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.150+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:36.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.150+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:36.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:36.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:36.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:36.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:36.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:36.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:36.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:36.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:36.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:36.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a3b0 con 0x7f63d8005e00 2021-03-09T10:37:36.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:36.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:36.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8005e00 2021-03-09T10:37:36.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:36.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:36.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.151+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18040 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f00365c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:36.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.152+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:37:36.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.152+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:36.165 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:36.154+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50150 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:36.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:37:36.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.155+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:36.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.155+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:36.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.155+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:36.167 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:36.167 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:36.167 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:36.168 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:36.168 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:36.168 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:36.168 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:36.168 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:36.169 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:36.169 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:36.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:37:36.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.157+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:36.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.157+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:37:36.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:37:36.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:36.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:36.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:37.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:37.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:37.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:37.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:38.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:38.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:38.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:38.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:39.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:39.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:39.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:39.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:40.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:40.610+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:40.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:40.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:40.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:40.676+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:41.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:41.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:41.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18040 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f00365c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:41.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18040 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f00365c0).stop 2021-03-09T10:37:41.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:41.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:41.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:41.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18040 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:41.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:37:41.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:41.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:41.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:41.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:41.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:41.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:41.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:41.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006be30 con 0x7f63d8003400 2021-03-09T10:37:41.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:41.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:37:41.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:41.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8003840 2021-03-09T10:37:41.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:41.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:41.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:41.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:41.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.155+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:41.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.156+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:41.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.156+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:37:41.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.156+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:41.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.156+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:41.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.157+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:41.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.157+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:41.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.157+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:41.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:41.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:41.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:41.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:41.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:41.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:37:41.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:37:41.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:41.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:41.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0000ab0 con 0x7f63d8003840 2021-03-09T10:37:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b560 con 0x7f63d8003840 2021-03-09T10:37:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:37:41.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:41.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:37:41.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:41.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.158+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 secure :-1 s=READY pgs=18057 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:41.172 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:41.162+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50188 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:41.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.162+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071170 con 0x7f63d8003840 2021-03-09T10:37:41.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.162+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:41.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.162+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:41.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.162+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:41.174 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:41.174 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:41.174 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:41.175 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:41.175 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:41.175 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:41.175 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:41.176 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:41.176 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:41.176 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:41.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.164+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071170 con 0x7f63d8003840 2021-03-09T10:37:41.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.164+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:41.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.166+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:37:41.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8003840 2021-03-09T10:37:41.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:41.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:41.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:42.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:42.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:42.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:42.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:43.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:43.675+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:43.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:43.675+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:44.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:44.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:44.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:44.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:45.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:45.611+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:45.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:45.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:45.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:45.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:46.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:37:46.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:46.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 secure :-1 s=READY pgs=18057 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:46.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 secure :-1 s=READY pgs=18057 cs=0 l=1 rev1=1 rx=0x7f63fc01deb0 tx=0x7f63fc009990).stop 2021-03-09T10:37:46.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:46.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:46.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:46.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015120 unknown :-1 s=CLOSED pgs=18057 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:46.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:37:46.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:46.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:46.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:46.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:46.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:46.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:46.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:46.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b5f0 con 0x7f63d8003400 2021-03-09T10:37:46.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:46.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001910 con 0x7f63d8009130 2021-03-09T10:37:46.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:46.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266f0 con 0x7f63d8005e00 2021-03-09T10:37:46.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:46.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:46.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:46.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:46.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:46.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.163+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:46.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:37:46.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:46.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:46.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:46.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:46.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:46.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:46.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:46.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:46.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:46.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:46.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:46.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:46.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:46.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:46.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001c640 con 0x7f63d8005e00 2021-03-09T10:37:46.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:46.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:46.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a3b0 con 0x7f63d8005e00 2021-03-09T10:37:46.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:46.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.164+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:46.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.165+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c150 secure :-1 s=READY pgs=18067 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0009ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:46.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.166+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:37:46.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.166+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:46.188 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:46.169+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50229 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:46.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00481f0 con 0x7f63d8005e00 2021-03-09T10:37:46.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.169+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:46.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.169+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:46.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.169+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:46.192 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:46.192 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:46.193 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:46.193 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:46.193 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:46.194 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:46.194 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:46.195 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:46.195 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:46.196 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:46.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8005e00 2021-03-09T10:37:46.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.172+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:46.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.172+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:37:46.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8005e00 2021-03-09T10:37:46.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:46.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:46.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:47.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:47.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:47.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:47.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:48.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:48.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:48.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:48.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:49.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:49.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:49.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:49.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:50.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:50.612+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:50.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:50.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:50.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:50.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:50.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:50.677+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:51.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:51.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:51.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c150 secure :-1 s=READY pgs=18067 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0009ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:51.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c150 secure :-1 s=READY pgs=18067 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0009ec0).stop 2021-03-09T10:37:51.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:51.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:51.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:51.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001c150 unknown :-1 s=CLOSED pgs=18067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:51.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:37:51.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:51.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:51.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:51.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8003400 2021-03-09T10:37:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:51.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8009130 2021-03-09T10:37:51.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:51.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00660b0 con 0x7f63d8003840 2021-03-09T10:37:51.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.169+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:51.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.170+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:51.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.170+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.170+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:51.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.170+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:51.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.171+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:51.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.171+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:37:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.171+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.171+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.171+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.171+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.171+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:51.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:51.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002fca0 con 0x7f63d8003840 2021-03-09T10:37:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001c640 con 0x7f63d8003840 2021-03-09T10:37:51.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.172+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.173+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 secure :-1 s=READY pgs=18082 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f005dec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.173+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:37:51.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.173+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:51.185 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:51.177+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50267 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:51.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00481f0 con 0x7f63d8003840 2021-03-09T10:37:51.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.177+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:51.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.177+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:51.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.177+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:51.186 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:51.186 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:51.187 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:51.187 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:51.187 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:51.187 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:51.188 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:51.188 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:51.188 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:51.188 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:51.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.181+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00483d0 con 0x7f63d8003840 2021-03-09T10:37:51.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.181+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:51.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.181+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:37:51.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.182+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8003840 2021-03-09T10:37:51.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:51.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:51.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:52.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:52.676+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:52.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:52.676+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:53.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:53.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:53.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:53.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:54.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:54.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:54.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:54.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:55.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:55.613+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:37:55.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:55.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:55.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:55.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:56.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:37:56.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:37:56.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 secure :-1 s=READY pgs=18082 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f005dec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:37:56.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 secure :-1 s=READY pgs=18082 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f005dec0).stop 2021-03-09T10:37:56.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:56.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:37:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:37:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:37:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:37:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:56.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:37:56.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:56.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:37:56.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:37:56.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:37:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8003400 2021-03-09T10:37:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:37:56.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:37:56.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b5f0 con 0x7f63d8005e00 2021-03-09T10:37:56.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:37:56.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:37:56.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:56.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:37:56.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.178+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:37:56.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.179+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:37:56.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.179+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:37:56.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.179+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:37:56.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.179+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:37:56.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.179+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:37:56.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.179+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:37:56.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.179+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:37:56.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:37:56.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:56.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:56.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:37:56.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:37:56.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:37:56.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:37:56.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:37:56.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:56.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010af0 con 0x7f63d8005e00 2021-03-09T10:37:56.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:37:56.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:37:56.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002fca0 con 0x7f63d8005e00 2021-03-09T10:37:56.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:37:56.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.180+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:56.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.181+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:37:56.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.181+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:37:56.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.181+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b560 secure :-1 s=READY pgs=18096 cs=0 l=1 rev1=1 rx=0x7f63f0055340 tx=0x7f63f0048440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:37:56.193 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:37:56.184+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50308 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:37:56.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:37:56.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.185+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:37:56.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.185+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:37:56.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.185+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:37:56.194 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:37:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:37:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:37:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:37:56.195 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:37:56.196 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:37:56.196 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:37:56.196 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:37:56.196 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:37:56.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:37:56.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.188+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:37:56.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.188+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:37:56.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.189+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8005e00 2021-03-09T10:37:56.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.677+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:56.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:56.677+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:57.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:57.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:57.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:58.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:58.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:58.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:58.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:37:59.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:59.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:37:59.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:37:59.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:00.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:00.614+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:00.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:00.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:00.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:00.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:00.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:00.678+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:01.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b560 secure :-1 s=READY pgs=18096 cs=0 l=1 rev1=1 rx=0x7f63f0055340 tx=0x7f63f0048440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b560 secure :-1 s=READY pgs=18096 cs=0 l=1 rev1=1 rx=0x7f63f0055340 tx=0x7f63f0048440).stop 2021-03-09T10:38:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:01.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001b560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001b560 unknown :-1 s=CLOSED pgs=18096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:38:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:01.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:01.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:01.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:01.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:01.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:01.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:01.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8003400 2021-03-09T10:38:01.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:01.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d9f0 con 0x7f63d8009130 2021-03-09T10:38:01.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:01.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0009530 con 0x7f63d8003840 2021-03-09T10:38:01.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:01.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:01.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:01.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:01.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.185+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:01.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.186+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:01.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.186+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:38:01.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.186+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:01.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.186+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:01.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:01.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:01.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:01.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:01.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:01.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:01.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:01.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:01.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007a810 con 0x7f63d8003840 2021-03-09T10:38:01.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:01.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:01.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010af0 con 0x7f63d8003840 2021-03-09T10:38:01.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:01.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.187+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:01.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.188+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:38:01.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.188+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:01.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.188+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 secure :-1 s=READY pgs=18103 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f0032e30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:01.200 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:01.192+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50346 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:01.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.193+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:38:01.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.193+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:01.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.193+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:01.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.193+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:01.202 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:01.202 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:01.202 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:01.202 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:01.203 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:01.203 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:01.203 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:01.203 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:01.203 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:01.204 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:01.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:38:01.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.196+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:01.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.196+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:38:01.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.198+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:38:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.678+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:01.678+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:02.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:02.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:02.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:02.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:03.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:03.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:03.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:03.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:04.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:04.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:04.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:04.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:05.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:05.615+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:05.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:05.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:05.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:05.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:06.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:38:06.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 secure :-1 s=READY pgs=18103 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f0032e30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 secure :-1 s=READY pgs=18103 cs=0 l=1 rev1=1 rx=0x7f63f007f7a0 tx=0x7f63f0032e30).stop 2021-03-09T10:38:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:06.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:06.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:06.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:38:06.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:06.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:06.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:06.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:06.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:06.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:38:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001af90 con 0x7f63d8009130 2021-03-09T10:38:06.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:06.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00660b0 con 0x7f63d8005e00 2021-03-09T10:38:06.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:06.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:06.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:06.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:06.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:38:06.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:06.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:06.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:06.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:06.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:06.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:06.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.194+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:06.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.195+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:06.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:06.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:06.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:06.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a3b0 con 0x7f63d8005e00 2021-03-09T10:38:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007a810 con 0x7f63d8005e00 2021-03-09T10:38:06.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:06.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.196+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:06.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.197+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18123 cs=0 l=1 rev1=1 rx=0x7f63f0055340 tx=0x7f63f004d110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:06.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.197+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:38:06.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.197+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:06.210 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:06.200+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50387 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:06.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.201+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:38:06.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.201+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:06.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.201+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:06.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.201+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:06.211 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:06.212 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:06.212 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:06.212 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:06.212 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:06.212 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:06.213 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:06.213 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:06.213 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:06.213 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:06.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.205+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:38:06.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.205+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:06.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.205+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:38:06.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.206+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:38:06.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:06.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:07.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:07.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:07.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:07.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:08.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:08.679+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:08.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:08.679+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:09.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:09.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:09.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:09.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:10.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:10.616+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:10.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:10.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:10.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:10.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:10.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:10.680+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:11.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:11.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:11.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18123 cs=0 l=1 rev1=1 rx=0x7f63f0055340 tx=0x7f63f004d110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:11.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18123 cs=0 l=1 rev1=1 rx=0x7f63f0055340 tx=0x7f63f004d110).stop 2021-03-09T10:38:11.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:11.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:11.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:11.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=CLOSED pgs=18123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:11.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:38:11.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:11.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:11.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:11.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:11.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:11.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:11.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:11.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:38:11.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266b0 con 0x7f63d8009130 2021-03-09T10:38:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8003840 2021-03-09T10:38:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.202+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:11.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.203+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.203+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.203+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:11.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.203+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.204+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.204+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:38:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.204+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:11.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.204+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.204+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.204+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:11.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:11.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:11.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:11.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:11.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065f00 con 0x7f63d8003840 2021-03-09T10:38:11.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a3b0 con 0x7f63d8003840 2021-03-09T10:38:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:11.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a200 secure :-1 s=READY pgs=18136 cs=0 l=1 rev1=1 rx=0x7f63f007a820 tx=0x7f63f0039410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:11.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:38:11.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.205+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:11.217 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:11.208+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50425 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:11.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0027e60 con 0x7f63d8003840 2021-03-09T10:38:11.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.209+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.209+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.209+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:11.219 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:11.220 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:11.221 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0027e60 con 0x7f63d8003840 2021-03-09T10:38:11.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.211+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.211+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:38:11.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.212+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:38:11.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:11.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:11.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:12.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:12.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:12.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:12.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:13.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:13.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:13.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:13.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:14.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:14.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:14.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:14.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:15.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:15.617+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:15.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:15.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:15.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:15.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:16.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:16.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:16.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a200 secure :-1 s=READY pgs=18136 cs=0 l=1 rev1=1 rx=0x7f63f007a820 tx=0x7f63f0039410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:16.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a200 secure :-1 s=READY pgs=18136 cs=0 l=1 rev1=1 rx=0x7f63f007a820 tx=0x7f63f0039410).stop 2021-03-09T10:38:16.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:16.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:16.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007a200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:16.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007a200 unknown :-1 s=CLOSED pgs=18136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:16.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:38:16.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:16.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:16.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:16.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:16.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:16.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:16.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:16.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8003400 2021-03-09T10:38:16.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:16.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d3c0 con 0x7f63d8009130 2021-03-09T10:38:16.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:16.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:38:16.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:16.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:16.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:16.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:16.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.210+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:16.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.210+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:16.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.210+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:38:16.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.210+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:16.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.210+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:16.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.211+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:16.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.211+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:16.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.211+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:16.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:16.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:16.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:16.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:16.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:16.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:16.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079a90 con 0x7f63d8005e00 2021-03-09T10:38:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:16.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:16.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065f00 con 0x7f63d8005e00 2021-03-09T10:38:16.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:16.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:16.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:38:16.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:16.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.212+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18148 cs=0 l=1 rev1=1 rx=0x7f63f005df50 tx=0x7f63f0013b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:16.224 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:16.216+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50466 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:16.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004c440 con 0x7f63d8005e00 2021-03-09T10:38:16.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.216+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:16.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.216+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:16.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.216+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:16.226 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:16.226 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:16.227 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:16.227 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:16.227 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:16.227 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:16.228 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:16.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004c260 con 0x7f63d8005e00 2021-03-09T10:38:16.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.216+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:16.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.216+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:38:16.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.218+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0086d20 con 0x7f63d8005e00 2021-03-09T10:38:16.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.680+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:16.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:16.680+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:17.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:17.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:17.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:17.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:18.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:18.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:18.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:18.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:19.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:19.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:19.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:19.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:20.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:20.618+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:20.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:20.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:20.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:20.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:20.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:20.681+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:21.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:21.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18148 cs=0 l=1 rev1=1 rx=0x7f63f005df50 tx=0x7f63f0013b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18148 cs=0 l=1 rev1=1 rx=0x7f63f005df50 tx=0x7f63f0013b90).stop 2021-03-09T10:38:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:21.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=CLOSED pgs=18148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:38:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:21.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:21.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:21.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:21.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:21.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:21.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:21.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b5f0 con 0x7f63d8003400 2021-03-09T10:38:21.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:21.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8009130 2021-03-09T10:38:21.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:38:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:21.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:21.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:21.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.217+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:21.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.217+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:38:21.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.217+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:21.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.217+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:21.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.217+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:21.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.217+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:21.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.217+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:21.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:21.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e840 con 0x7f63d8003840 2021-03-09T10:38:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:21.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079a90 con 0x7f63d8003840 2021-03-09T10:38:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.218+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.219+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e350 secure :-1 s=READY pgs=18162 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f004c2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.219+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:38:21.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.219+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:21.475 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:21.223+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50504 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:38:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.224+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.224+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:21.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.224+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:21.477 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:21.477 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:21.477 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:21.477 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:21.478 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:21.478 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:21.478 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:21.478 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:21.478 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:21.479 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:38:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.226+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:21.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.226+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:38:21.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.227+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0086d20 con 0x7f63d8003840 2021-03-09T10:38:21.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:21.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:21.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:22.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:22.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:22.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:23.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:23.681+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:23.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:23.681+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:24.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:24.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:24.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:24.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:25.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:25.619+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:25.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:25.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:25.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:25.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:26.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:26.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:26.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e350 secure :-1 s=READY pgs=18162 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f004c2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:26.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e350 secure :-1 s=READY pgs=18162 cs=0 l=1 rev1=1 rx=0x7f63f0005010 tx=0x7f63f004c2d0).stop 2021-03-09T10:38:26.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e350 unknown :-1 s=CLOSED pgs=18162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:38:26.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:26.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:26.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:26.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:26.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026e20 con 0x7f63d8003400 2021-03-09T10:38:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:26.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63d8009130 2021-03-09T10:38:26.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:26.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8005e00 2021-03-09T10:38:26.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:26.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:26.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.224+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:26.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.225+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:26.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.225+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:26.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.225+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:26.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.225+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:38:26.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.225+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:26.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.225+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:26.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:26.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:26.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:26.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:26.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:26.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:26.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:26.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:26.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002d50 con 0x7f63d8005e00 2021-03-09T10:38:26.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:26.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:26.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e840 con 0x7f63d8005e00 2021-03-09T10:38:26.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:26.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.226+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:26.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.227+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18177 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f00313d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:26.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.227+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:38:26.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.227+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:26.238 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:26.231+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50545 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:26.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8005e00 2021-03-09T10:38:26.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.232+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.232+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.232+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:26.240 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:26.241 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:26.241 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:26.241 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:26.241 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:26.241 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:26.242 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:26.242 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:26.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8005e00 2021-03-09T10:38:26.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.234+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:26.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.235+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:38:26.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:38:26.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:26.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:26.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:27.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:27.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:27.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:27.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:28.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:28.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:28.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:29.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:29.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:29.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:29.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:30.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:30.620+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:30.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:30.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:30.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:30.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:30.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:30.682+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:31.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:31.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:31.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18177 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f00313d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:31.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18177 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f00313d0).stop 2021-03-09T10:38:31.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:31.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:31.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:31.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=CLOSED pgs=18177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:31.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:38:31.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:31.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:31.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:31.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:31.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:31.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:31.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:31.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:38:31.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:31.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:38:31.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8003840 2021-03-09T10:38:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.231+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:31.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.232+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:38:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.232+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:31.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.232+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:31.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.232+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:31.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.232+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:31.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.232+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:31.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:31.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:31.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:31.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:31.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:31.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:31.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:31.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:31.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:31.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:38:31.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:31.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:31.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e840 con 0x7f63d8003840 2021-03-09T10:38:31.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:31.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.233+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:31.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.234+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 secure :-1 s=READY pgs=18188 cs=0 l=1 rev1=1 rx=0x7f63f0055340 tx=0x7f63f0006680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:31.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.234+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:38:31.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.234+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:31.246 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:31.238+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50583 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:31.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.241+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005fcb0 con 0x7f63d8003840 2021-03-09T10:38:31.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.241+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:31.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.241+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:31.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.241+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:31.247 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:31.247 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:31.248 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:31.248 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:31.248 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:31.248 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:31.248 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:31.249 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:31.249 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:31.249 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:31.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005fcb0 con 0x7f63d8003840 2021-03-09T10:38:31.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.243+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:31.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.243+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:38:31.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8003840 2021-03-09T10:38:31.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:31.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:31.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:32.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:32.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:32.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:32.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:33.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:33.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:33.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:33.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:34.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:34.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:34.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:34.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:35.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:35.620+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:35.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:35.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:35.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:35.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:36.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.237+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:36.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.237+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:36.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.237+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 secure :-1 s=READY pgs=18188 cs=0 l=1 rev1=1 rx=0x7f63f0055340 tx=0x7f63f0006680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:36.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.237+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 secure :-1 s=READY pgs=18188 cs=0 l=1 rev1=1 rx=0x7f63f0055340 tx=0x7f63f0006680).stop 2021-03-09T10:38:36.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:36.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:36.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ba10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:36.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 unknown :-1 s=CLOSED pgs=18188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:36.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:38:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:36.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:36.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:36.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:36.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d2a0 con 0x7f63d8003400 2021-03-09T10:38:36.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:36.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013d00 con 0x7f63d8009130 2021-03-09T10:38:36.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:36.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d3c0 con 0x7f63d8005e00 2021-03-09T10:38:36.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:36.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:36.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:38:36.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:36.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.238+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:36.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:36.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:36.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:36.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:36.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:36.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:36.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:36.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:36.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:36.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:36.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:36.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:36.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:38:36.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:38:36.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:36.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:36.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00240e0 con 0x7f63d8005e00 2021-03-09T10:38:36.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:36.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:36.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0000ab0 con 0x7f63d8005e00 2021-03-09T10:38:36.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:38:36.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.239+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:36.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.240+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 secure :-1 s=READY pgs=18201 cs=0 l=1 rev1=1 rx=0x7f63e006af30 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:36.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.240+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:38:36.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.240+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:36.254 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:36.246+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50633 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:36.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.246+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01fba0 con 0x7f63d8005e00 2021-03-09T10:38:36.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.246+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:36.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.246+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.246+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:36.256 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:36.257 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:36.257 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:36.257 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:36.257 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:36.257 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:36.258 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:36.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.246+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01fba0 con 0x7f63d8005e00 2021-03-09T10:38:36.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.246+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:36.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.246+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:38:36.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.248+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8005e00 2021-03-09T10:38:36.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:36.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:36.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:37.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:37.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:37.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:37.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:38.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:38.682+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:38.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:38.682+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:39.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:39.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:39.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:39.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:40.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:40.621+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:40.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:40.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:40.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:40.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:40.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:40.683+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:41.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 secure :-1 s=READY pgs=18201 cs=0 l=1 rev1=1 rx=0x7f63e006af30 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 secure :-1 s=READY pgs=18201 cs=0 l=1 rev1=1 rx=0x7f63e006af30 tx=0x7f63fc003a50).stop 2021-03-09T10:38:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:41.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 unknown :-1 s=CLOSED pgs=18201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:38:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:41.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:41.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:41.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:41.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:41.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:41.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:41.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8003400 2021-03-09T10:38:41.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:41.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:38:41.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:41.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:38:41.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:41.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:41.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:41.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:41.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.246+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:41.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.247+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:41.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.247+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:38:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.247+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.247+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.248+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.248+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:41.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.248+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:41.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:41.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:41.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:41.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:41.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:41.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:41.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:41.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:41.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:41.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0000ab0 con 0x7f63d8003840 2021-03-09T10:38:41.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:41.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:41.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:38:41.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:41.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:41.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=18220 cs=0 l=1 rev1=1 rx=0x7f63f0089070 tx=0x7f63f0011720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:41.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:38:41.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.249+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:41.261 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:41.253+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50671 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:41.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.258+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:38:41.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.258+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:41.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.258+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:41.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.258+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:41.262 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:41.262 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:41.263 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:41.263 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:41.263 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:41.263 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:41.264 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:41.264 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:41.264 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:41.264 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:41.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.258+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00118e0 con 0x7f63d8003840 2021-03-09T10:38:41.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.258+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:41.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.258+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0011da0 con 0x7f63d8003840 2021-03-09T10:38:41.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.258+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:38:41.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:41.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:41.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:42.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:42.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:42.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:42.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:43.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:43.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:43.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:43.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:44.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:44.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:44.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:44.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:45.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:45.622+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:45.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:45.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:45.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:45.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:46.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.253+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:46.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.253+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:46.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.253+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=18220 cs=0 l=1 rev1=1 rx=0x7f63f0089070 tx=0x7f63f0011720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:46.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=18220 cs=0 l=1 rev1=1 rx=0x7f63f0089070 tx=0x7f63f0011720).stop 2021-03-09T10:38:46.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:46.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:46.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:46.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=CLOSED pgs=18220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:46.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:38:46.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:46.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:46.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:46.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:46.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266b0 con 0x7f63d8003400 2021-03-09T10:38:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d110 con 0x7f63d8009130 2021-03-09T10:38:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ad30 con 0x7f63d8005e00 2021-03-09T10:38:46.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:46.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:46.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:46.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:46.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:46.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:46.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.254+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:38:46.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.255+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:46.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.255+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:46.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.255+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:46.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.255+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:46.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.255+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:46.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:46.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:46.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:46.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:46.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:46.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0008ee0 con 0x7f63d8005e00 2021-03-09T10:38:46.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:46.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:46.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0000ab0 con 0x7f63d8005e00 2021-03-09T10:38:46.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:46.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.256+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:46.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.257+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005cba0 secure :-1 s=READY pgs=18233 cs=0 l=1 rev1=1 rx=0x7f63e002b9c0 tx=0x7f63f0003520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:46.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.257+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:38:46.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.257+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:46.269 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:46.260+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50712 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:46.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:38:46.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.261+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:46.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.261+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:46.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.261+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:46.272 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:46.273 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:46.273 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:46.273 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:46.273 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:46.274 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:46.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.262+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:38:46.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.262+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:46.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.262+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:38:46.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.263+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8005e00 2021-03-09T10:38:46.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.683+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:46.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:46.683+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:47.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:47.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:47.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:47.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:48.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:48.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:48.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:48.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:49.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:49.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:49.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:49.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:50.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:50.623+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:50.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:50.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:50.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:50.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:50.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:50.684+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:51.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.260+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:38:51.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.260+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e005cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:51.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.260+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005cba0 secure :-1 s=READY pgs=18233 cs=0 l=1 rev1=1 rx=0x7f63e002b9c0 tx=0x7f63f0003520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:51.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.260+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005cba0 secure :-1 s=READY pgs=18233 cs=0 l=1 rev1=1 rx=0x7f63e002b9c0 tx=0x7f63f0003520).stop 2021-03-09T10:38:51.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:51.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:51.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e005cba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:51.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e005cba0 unknown :-1 s=CLOSED pgs=18233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:51.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:38:51.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:51.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:51.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:51.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:51.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:51.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:51.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:51.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8003400 2021-03-09T10:38:51.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:51.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:38:51.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:51.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023fa0 con 0x7f63d8003840 2021-03-09T10:38:51.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:51.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:51.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:51.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:51.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:51.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:51.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:38:51.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:51.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:51.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:51.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:51.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.261+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:51.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:51.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:51.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:51.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:51.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:51.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:51.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:51.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:51.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:51.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0000ab0 con 0x7f63d8003840 2021-03-09T10:38:51.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:51.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:51.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0008ee0 con 0x7f63d8003840 2021-03-09T10:38:51.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:51.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:51.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=18244 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f0073fc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:51.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:38:51.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.263+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:51.276 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:51.267+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50750 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:51.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.267+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005feb0 con 0x7f63d8003840 2021-03-09T10:38:51.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.267+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:51.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.267+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:51.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.267+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:51.278 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:51.278 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:51.278 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:51.279 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:51.279 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:51.279 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:51.279 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:51.279 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:51.280 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:51.280 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:51.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.268+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005fcb0 con 0x7f63d8003840 2021-03-09T10:38:51.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.268+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:51.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.268+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:38:51.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.271+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0073340 con 0x7f63d8003840 2021-03-09T10:38:51.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:51.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:51.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:52.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:52.684+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:52.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:52.684+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:53.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:53.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:53.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:53.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:54.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:54.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:54.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:54.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:55.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:55.624+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:38:55.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:55.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:55.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:55.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:56.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.267+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:38:56.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.267+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:38:56.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.267+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=18244 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f0073fc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:38:56.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.267+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=18244 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f0073fc0).stop 2021-03-09T10:38:56.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:56.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:38:56.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:38:56.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=CLOSED pgs=18244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:38:56.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:38:56.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:56.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:38:56.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:56.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:38:56.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:38:56.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:38:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:56.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077340 con 0x7f63d8003400 2021-03-09T10:38:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:56.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d3c0 con 0x7f63d8009130 2021-03-09T10:38:56.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:38:56.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:38:56.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:38:56.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:38:56.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:38:56.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.268+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:38:56.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.269+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:38:56.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.269+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:38:56.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.269+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:38:56.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.269+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:38:56.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.269+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:38:56.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.269+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:38:56.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.269+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:38:56.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:38:56.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:56.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:56.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:38:56.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:38:56.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:38:56.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:38:56.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:38:56.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:56.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007af20 con 0x7f63d8005e00 2021-03-09T10:38:56.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:38:56.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:38:56.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0000ab0 con 0x7f63d8005e00 2021-03-09T10:38:56.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:38:56.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:56.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 secure :-1 s=READY pgs=18257 cs=0 l=1 rev1=1 rx=0x7f63f0060070 tx=0x7f63f00393a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:38:56.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:38:56.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.270+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:38:56.304 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:38:56.275+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50791 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:38:56.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.276+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:38:56.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.276+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:38:56.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.276+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:38:56.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.276+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:38:56.305 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:38:56.306 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:38:56.306 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:56.306 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:38:56.306 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:38:56.307 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:38:56.307 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:38:56.307 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:38:56.307 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:38:56.307 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:38:56.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.277+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:38:56.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.277+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:38:56.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.277+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:38:56.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.278+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8005e00 2021-03-09T10:38:56.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:56.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:56.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:57.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:57.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:57.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:57.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:58.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:58.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:38:58.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:58.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:38:59.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:59.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:00.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:38:59.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:00.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:00.625+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:00.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:00.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:00.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:00.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:00.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:00.685+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:01.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:01.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:01.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 secure :-1 s=READY pgs=18257 cs=0 l=1 rev1=1 rx=0x7f63f0060070 tx=0x7f63f00393a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:01.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 secure :-1 s=READY pgs=18257 cs=0 l=1 rev1=1 rx=0x7f63f0060070 tx=0x7f63f00393a0).stop 2021-03-09T10:39:01.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:01.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:01.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006d020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:01.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006d020 unknown :-1 s=CLOSED pgs=18257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:01.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:39:01.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:01.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:01.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:01.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:01.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:01.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:01.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:01.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8003400 2021-03-09T10:39:01.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:01.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:39:01.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:01.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021730 con 0x7f63d8003840 2021-03-09T10:39:01.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.275+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:01.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.276+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:01.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.276+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:01.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.276+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:01.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.276+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:01.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.277+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:01.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.277+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:39:01.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.277+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:01.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.277+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:01.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.277+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:01.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.277+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:01.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.277+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:01.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:01.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:01.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:01.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:01.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:01.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000b9a0 con 0x7f63d8003840 2021-03-09T10:39:01.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007af20 con 0x7f63d8003840 2021-03-09T10:39:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:01.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e450 secure :-1 s=READY pgs=18275 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0042400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:01.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:39:01.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.279+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:01.302 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:01.283+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50829 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:01.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:39:01.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.284+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:01.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.284+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:01.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.284+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:01.304 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:01.304 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:01.304 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:01.305 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:01.305 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:01.305 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:01.305 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:01.305 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:01.306 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:01.306 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:01.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.285+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:39:01.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.285+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:01.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.285+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:39:01.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8003840 2021-03-09T10:39:01.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:01.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:01.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:02.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:02.685+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:02.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:02.685+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:03.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:03.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:03.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:03.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:04.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:04.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:04.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:04.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:05.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:05.626+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:05.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:05.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:05.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:05.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:06.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:06.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:06.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e450 secure :-1 s=READY pgs=18275 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0042400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:06.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e450 secure :-1 s=READY pgs=18275 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0042400).stop 2021-03-09T10:39:06.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:06.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:06.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002e450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:06.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e450 unknown :-1 s=CLOSED pgs=18275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:06.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:39:06.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:06.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:06.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:06.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d3c0 con 0x7f63d8003400 2021-03-09T10:39:06.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:06.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012a30 con 0x7f63d8009130 2021-03-09T10:39:06.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:06.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:39:06.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:06.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:06.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:06.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:06.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.284+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:06.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.285+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:06.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.285+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:39:06.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.285+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:06.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.285+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:06.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.285+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:06.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.285+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:06.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.285+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:06.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:06.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:06.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:06.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:06.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:06.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:06.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:06.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:06.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:06.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63d8005e00 2021-03-09T10:39:06.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:06.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:06.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000b9a0 con 0x7f63d8005e00 2021-03-09T10:39:06.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:06.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:06.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:39:06.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:06.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.286+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006da10 secure :-1 s=READY pgs=18285 cs=0 l=1 rev1=1 rx=0x7f63f007a5b0 tx=0x7f63f004d230).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:06.304 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:06.293+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50870 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:06.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0029670 con 0x7f63d8005e00 2021-03-09T10:39:06.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.293+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:06.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.293+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:06.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.293+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:06.306 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:06.306 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:06.306 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:06.307 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:06.307 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:06.307 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:06.307 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:06.307 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:06.308 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:06.308 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:06.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.294+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0029840 con 0x7f63d8005e00 2021-03-09T10:39:06.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.294+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:06.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.294+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:39:06.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63d8005e00 2021-03-09T10:39:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:06.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:07.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:07.686+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:07.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:07.686+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:08.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:08.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:08.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:08.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:09.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:09.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:09.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:09.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:10.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:10.627+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:10.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:10.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:10.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:10.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:10.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:10.687+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:11.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006da10 secure :-1 s=READY pgs=18285 cs=0 l=1 rev1=1 rx=0x7f63f007a5b0 tx=0x7f63f004d230).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006da10 secure :-1 s=READY pgs=18285 cs=0 l=1 rev1=1 rx=0x7f63f007a5b0 tx=0x7f63f004d230).stop 2021-03-09T10:39:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:11.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006da10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006da10 unknown :-1 s=CLOSED pgs=18285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:39:11.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:11.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8003400 2021-03-09T10:39:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:39:11.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:11.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8003840 2021-03-09T10:39:11.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.293+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:11.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:11.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:11.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:11.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:11.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:11.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:39:11.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:11.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:11.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:11.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:11.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.294+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:11.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:11.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:11.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:11.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:11.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:11.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:39:11.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:39:11.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:11.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce10 con 0x7f63d8003840 2021-03-09T10:39:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00240e0 con 0x7f63d8003840 2021-03-09T10:39:11.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:39:11.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.295+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:11.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.296+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 secure :-1 s=READY pgs=18298 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc03ffd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:11.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.296+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:39:11.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.296+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:11.308 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:11.300+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50908 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:11.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058000 con 0x7f63d8003840 2021-03-09T10:39:11.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.300+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:11.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.300+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:11.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.300+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:11.309 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:11.310 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:11.310 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:11.310 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:11.310 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:11.311 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:11.311 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:11.311 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:11.311 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:11.311 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:11.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01fba0 con 0x7f63d8003840 2021-03-09T10:39:11.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.305+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:11.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01fd10 con 0x7f63d8003840 2021-03-09T10:39:11.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.305+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:39:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:11.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:12.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:12.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:12.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:12.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:13.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:13.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:13.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:13.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:14.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:14.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:14.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:14.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:15.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:15.628+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:15.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:15.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:15.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:15.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:16.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:16.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 secure :-1 s=READY pgs=18298 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc03ffd0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 secure :-1 s=READY pgs=18298 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc03ffd0).stop 2021-03-09T10:39:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:16.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:16.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 unknown :-1 s=CLOSED pgs=18298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:16.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:39:16.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:16.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:16.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:16.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:16.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:16.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:16.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077340 con 0x7f63d8003400 2021-03-09T10:39:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021730 con 0x7f63d8009130 2021-03-09T10:39:16.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d3c0 con 0x7f63d8005e00 2021-03-09T10:39:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.300+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:16.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.301+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:16.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:16.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.301+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:16.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.301+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:39:16.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.301+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:16.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.301+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:16.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:16.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:16.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.302+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:16.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:16.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:16.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:16.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:16.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:16.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:16.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:16.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:16.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:16.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001e390 con 0x7f63d8005e00 2021-03-09T10:39:16.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:16.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:16.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63d8005e00 2021-03-09T10:39:16.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:16.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:16.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18314 cs=0 l=1 rev1=1 rx=0x7f63f0039370 tx=0x7f63f0006610).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:16.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:39:16.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.303+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:16.315 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:16.309+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50949 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:16.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.310+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8005e00 2021-03-09T10:39:16.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.310+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:16.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.310+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:16.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.310+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:16.317 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:16.317 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:16.318 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:16.318 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:16.318 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:16.318 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:16.319 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:16.319 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:16.319 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:16.319 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:16.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.312+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005df20 con 0x7f63d8005e00 2021-03-09T10:39:16.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.312+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:16.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.312+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:39:16.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8005e00 2021-03-09T10:39:16.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.687+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:16.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:16.687+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:17.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:17.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:17.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:18.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:18.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:18.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:18.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:19.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:19.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:19.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:19.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:20.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:20.629+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:20.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:20.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:20.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:20.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:20.688+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:21.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:21.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:21.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18314 cs=0 l=1 rev1=1 rx=0x7f63f0039370 tx=0x7f63f0006610).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:21.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18314 cs=0 l=1 rev1=1 rx=0x7f63f0039370 tx=0x7f63f0006610).stop 2021-03-09T10:39:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=CLOSED pgs=18314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:21.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:39:21.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:21.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:21.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:21.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:21.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:21.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:21.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:21.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:39:21.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:21.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012a30 con 0x7f63d8009130 2021-03-09T10:39:21.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:21.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8003840 2021-03-09T10:39:21.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:21.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:21.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:21.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:21.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.309+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:21.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.310+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:21.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.310+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:39:21.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.310+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:21.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.310+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:21.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.310+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:21.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.310+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:21.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.310+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:21.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:21.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:21.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:21.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:21.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:21.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:21.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:21.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:21.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:21.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0023790 con 0x7f63d8003840 2021-03-09T10:39:21.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:21.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:21.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001e390 con 0x7f63d8003840 2021-03-09T10:39:21.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:21.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.311+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:21.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.312+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 secure :-1 s=READY pgs=18326 cs=0 l=1 rev1=1 rx=0x7f63f00118c0 tx=0x7f63f0030300).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:21.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.312+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:39:21.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.312+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:21.325 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:21.317+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 50987 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:21.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:39:21.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.323+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:21.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.323+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:21.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.323+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:21.326 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:21.326 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:21.327 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:21.327 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:21.327 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:21.327 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:21.328 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:21.328 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:21.328 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:21.328 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:21.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.323+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005dbb0 con 0x7f63d8003840 2021-03-09T10:39:21.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.323+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:21.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.323+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:39:21.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:39:21.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:21.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:21.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:22.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:22.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:23.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:23.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:23.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:23.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:24.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:24.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:24.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:24.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:25.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:25.630+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:25.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:25.688+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:25.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:25.688+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:26.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:26.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:26.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 secure :-1 s=READY pgs=18326 cs=0 l=1 rev1=1 rx=0x7f63f00118c0 tx=0x7f63f0030300).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:26.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 secure :-1 s=READY pgs=18326 cs=0 l=1 rev1=1 rx=0x7f63f00118c0 tx=0x7f63f0030300).stop 2021-03-09T10:39:26.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:26.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:26.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:26.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cf0 unknown :-1 s=CLOSED pgs=18326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:26.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:39:26.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:26.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:26.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:26.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:26.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:26.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:26.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:26.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:39:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017fd0 con 0x7f63d8009130 2021-03-09T10:39:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de20 con 0x7f63d8005e00 2021-03-09T10:39:26.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:26.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:26.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.317+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:26.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.318+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:26.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.318+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:26.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.318+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:26.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.318+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:39:26.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.318+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:26.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.318+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:26.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.318+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:26.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.318+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:26.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.318+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:26.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:26.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:26.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:26.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:26.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:26.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:26.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:26.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:26.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:26.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022df0 con 0x7f63d8005e00 2021-03-09T10:39:26.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:26.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:26.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0023790 con 0x7f63d8005e00 2021-03-09T10:39:26.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:26.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:26.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18339 cs=0 l=1 rev1=1 rx=0x7f63f0060520 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:26.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:39:26.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.319+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:26.332 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:26.325+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51028 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:26.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0073340 con 0x7f63d8005e00 2021-03-09T10:39:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.325+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.325+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:26.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.325+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:26.335 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:26.336 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:26.337 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:26.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007ee30 con 0x7f63d8005e00 2021-03-09T10:39:26.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.325+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:26.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.325+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:39:26.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.326+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8005e00 2021-03-09T10:39:26.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:26.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:26.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:27.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:27.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:27.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:27.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:28.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:28.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:28.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:28.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:29.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:29.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:29.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:29.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:30.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:30.631+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:30.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:30.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:30.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:30.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:30.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:30.689+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:31.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:31.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:31.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18339 cs=0 l=1 rev1=1 rx=0x7f63f0060520 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:31.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 secure :-1 s=READY pgs=18339 cs=0 l=1 rev1=1 rx=0x7f63f0060520 tx=0x7f63f0000c30).stop 2021-03-09T10:39:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00164d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:31.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00164d0 unknown :-1 s=CLOSED pgs=18339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:39:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:31.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:31.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:31.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:31.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:31.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:31.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007af70 con 0x7f63d8003400 2021-03-09T10:39:31.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:31.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021730 con 0x7f63d8009130 2021-03-09T10:39:31.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:31.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8003840 2021-03-09T10:39:31.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:31.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:31.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.325+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:31.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.326+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:31.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.326+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:31.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.327+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:31.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.327+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:39:31.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.327+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:31.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.327+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:31.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:31.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:31.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:31.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:31.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:31.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:31.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:31.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:31.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:31.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:31.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:31.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:31.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018110 con 0x7f63d8003840 2021-03-09T10:39:31.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022df0 con 0x7f63d8003840 2021-03-09T10:39:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:31.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a540 secure :-1 s=READY pgs=18360 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:31.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:39:31.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.329+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:31.342 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:31.332+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51066 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:31.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:39:31.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.333+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:31.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.333+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:31.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.333+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:31.344 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:31.345 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:31.345 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:31.345 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:31.345 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:31.345 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:31.346 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:31.346 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:31.346 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:31.347 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:31.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.337+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:39:31.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.338+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:31.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.338+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:39:31.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.338+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005dbb0 con 0x7f63d8003840 2021-03-09T10:39:31.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.689+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:31.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:31.689+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:32.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:32.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:32.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:32.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:33.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:33.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:33.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:33.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:34.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:34.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:34.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:34.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:35.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:35.632+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:35.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:35.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:35.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:35.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:36.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:36.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:36.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a540 secure :-1 s=READY pgs=18360 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:36.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a540 secure :-1 s=READY pgs=18360 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004d180).stop 2021-03-09T10:39:36.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:36.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:36.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002a540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:36.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a540 unknown :-1 s=CLOSED pgs=18360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:36.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:39:36.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:36.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:36.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:36.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:36.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:36.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:36.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:36.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077340 con 0x7f63d8003400 2021-03-09T10:39:36.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:36.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8009130 2021-03-09T10:39:36.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:36.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:39:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:36.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.333+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:39:36.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:36.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:36.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:36.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:36.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:36.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:36.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:36.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065ec0 con 0x7f63d8005e00 2021-03-09T10:39:36.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:36.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:36.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018110 con 0x7f63d8005e00 2021-03-09T10:39:36.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:36.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.334+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:36.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.335+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18366 cs=0 l=1 rev1=1 rx=0x7f63f0061020 tx=0x7f63f0036de0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:36.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.335+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:39:36.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.335+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:36.349 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:36.339+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51107 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f650 con 0x7f63d8005e00 2021-03-09T10:39:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.340+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.340+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:36.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.340+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:36.351 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:36.352 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:36.352 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028c70 con 0x7f63d8005e00 2021-03-09T10:39:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.340+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.340+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:39:36.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.341+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8005e00 2021-03-09T10:39:36.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:36.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:36.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:37.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:37.690+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:37.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:37.690+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:38.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:38.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:38.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:38.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:39.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:39.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:39.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:39.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:40.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:40.633+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:40.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:40.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:40.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:40.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:40.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:40.691+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:41.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:39:41.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:41.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18366 cs=0 l=1 rev1=1 rx=0x7f63f0061020 tx=0x7f63f0036de0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:41.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18366 cs=0 l=1 rev1=1 rx=0x7f63f0061020 tx=0x7f63f0036de0).stop 2021-03-09T10:39:41.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:41.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:41.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:41.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=18366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:41.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:39:41.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:41.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:41.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:41.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:41.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:41.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:41.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:41.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028700 con 0x7f63d8003400 2021-03-09T10:39:41.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:41.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bac0 con 0x7f63d8009130 2021-03-09T10:39:41.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:41.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003840 2021-03-09T10:39:41.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:41.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:41.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:41.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:41.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:41.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:41.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:39:41.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:41.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.340+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:41.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:41.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:41.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:41.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:41.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:41.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:41.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:41.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:41.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:39:41.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:39:41.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:41.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:41.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026c20 con 0x7f63d8003840 2021-03-09T10:39:41.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:41.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:41.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce10 con 0x7f63d8003840 2021-03-09T10:39:41.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:39:41.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:41.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:39:41.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ad30 secure :-1 s=READY pgs=18380 cs=0 l=1 rev1=1 rx=0x7f63e0001f50 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:41.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.341+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:41.355 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:41.348+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51145 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:41.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02b5e0 con 0x7f63d8003840 2021-03-09T10:39:41.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.349+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:41.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.349+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:41.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.349+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:41.357 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:41.357 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:41.357 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:41.358 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:41.358 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:41.358 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:41.358 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:41.358 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:41.359 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:41.359 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:41.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:39:41.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.349+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:41.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.349+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:39:41.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.350+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f63d8003840 2021-03-09T10:39:41.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:41.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:41.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:42.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:42.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:42.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:42.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:43.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:43.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:43.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:43.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:44.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:44.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:44.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:44.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:45.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:45.634+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:45.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:45.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:45.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:45.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:46.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.348+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:46.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.348+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:46.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.348+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ad30 secure :-1 s=READY pgs=18380 cs=0 l=1 rev1=1 rx=0x7f63e0001f50 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:46.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.348+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ad30 secure :-1 s=READY pgs=18380 cs=0 l=1 rev1=1 rx=0x7f63e0001f50 tx=0x7f63fc009990).stop 2021-03-09T10:39:46.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.348+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:46.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:46.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ad30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:46.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ad30 unknown :-1 s=CLOSED pgs=18380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:46.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:39:46.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:46.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:46.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:46.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:46.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:46.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:46.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:46.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fe20 con 0x7f63d8003400 2021-03-09T10:39:46.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:46.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012740 con 0x7f63d8009130 2021-03-09T10:39:46.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:46.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077340 con 0x7f63d8005e00 2021-03-09T10:39:46.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:46.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:46.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:39:46.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:46.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:46.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:46.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:46.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.349+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:46.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:46.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:46.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:46.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:46.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:46.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:46.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:46.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:46.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:46.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016d60 con 0x7f63d8005e00 2021-03-09T10:39:46.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:46.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:46.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065ec0 con 0x7f63d8005e00 2021-03-09T10:39:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.350+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=18393 cs=0 l=1 rev1=1 rx=0x7f63e0025150 tx=0x7f63f004dab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.351+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:39:46.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.351+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:46.364 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:46.355+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51186 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:46.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f650 con 0x7f63d8005e00 2021-03-09T10:39:46.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.356+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:46.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.356+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:46.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.356+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:46.365 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:46.366 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:46.366 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:46.366 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:46.366 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:46.367 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:46.367 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:46.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.359+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007f650 con 0x7f63d8005e00 2021-03-09T10:39:46.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.359+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:46.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.359+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:39:46.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.360+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0012350 con 0x7f63d8005e00 2021-03-09T10:39:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.691+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:46.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:46.691+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:47.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:47.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:47.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:47.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:48.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:48.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:48.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:48.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:49.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:49.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:49.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:49.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:50.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:50.635+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:50.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:50.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:50.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:50.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:50.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:50.692+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:51.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:51.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:51.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=18393 cs=0 l=1 rev1=1 rx=0x7f63e0025150 tx=0x7f63f004dab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:51.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=18393 cs=0 l=1 rev1=1 rx=0x7f63e0025150 tx=0x7f63f004dab0).stop 2021-03-09T10:39:51.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:51.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:51.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:51.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=CLOSED pgs=18393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:39:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:51.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:51.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:51.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:51.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:51.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c010 con 0x7f63d8003400 2021-03-09T10:39:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:39:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065ce0 con 0x7f63d8003840 2021-03-09T10:39:51.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:51.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:51.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:51.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:51.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.356+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.357+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.357+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:39:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.357+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:51.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.357+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:51.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.357+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:51.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.357+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:51.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.357+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:51.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:51.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:51.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:51.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bda0 con 0x7f63d8003840 2021-03-09T10:39:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:51.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016d60 con 0x7f63d8003840 2021-03-09T10:39:51.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:51.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:51.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 secure :-1 s=READY pgs=18407 cs=0 l=1 rev1=1 rx=0x7f63f0088640 tx=0x7f63f00883c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:51.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:39:51.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.358+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:51.371 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:51.364+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51224 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0088ef0 con 0x7f63d8003840 2021-03-09T10:39:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.364+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:51.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.364+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.364+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:51.373 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:51.374 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:51.375 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0088ef0 con 0x7f63d8003840 2021-03-09T10:39:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.364+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.364+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:39:51.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.365+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:39:51.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:51.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:51.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:52.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:52.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:52.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:53.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:53.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:53.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:53.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:54.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:54.692+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:54.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:54.692+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:55.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:55.636+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:39:55.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:55.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:55.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:55.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:56.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:39:56.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:39:56.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 secure :-1 s=READY pgs=18407 cs=0 l=1 rev1=1 rx=0x7f63f0088640 tx=0x7f63f00883c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:39:56.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 secure :-1 s=READY pgs=18407 cs=0 l=1 rev1=1 rx=0x7f63f0088640 tx=0x7f63f00883c0).stop 2021-03-09T10:39:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:39:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:39:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 unknown :-1 s=CLOSED pgs=18407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:39:56.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:39:56.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:56.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:39:56.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:56.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:39:56.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:39:56.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:39:56.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:56.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:39:56.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077340 con 0x7f63d8009130 2021-03-09T10:39:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:39:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0019ce0 con 0x7f63d8005e00 2021-03-09T10:39:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.364+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:39:56.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.365+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:39:56.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.365+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:56.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.365+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:39:56.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.365+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:39:56.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.366+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:39:56.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.366+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:39:56.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.366+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:39:56.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.366+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:39:56.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.366+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:39:56.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.366+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:39:56.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.366+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:39:56.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.367+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:39:56.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.367+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:56.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.367+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:56.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:39:56.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:39:56.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:39:56.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:39:56.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:39:56.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:56.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065ec0 con 0x7f63d8005e00 2021-03-09T10:39:56.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:39:56.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:39:56.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bda0 con 0x7f63d8005e00 2021-03-09T10:39:56.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:39:56.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:56.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:39:56.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:39:56.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.368+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=18431 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f005d560).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:39:56.380 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:39:56.372+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51265 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:39:56.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.373+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:39:56.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.373+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:39:56.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.373+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:39:56.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.373+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:39:56.382 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:39:56.382 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:39:56.382 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:56.382 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:39:56.382 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:39:56.383 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:39:56.383 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:39:56.383 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:39:56.383 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:39:56.383 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:39:56.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8005e00 2021-03-09T10:39:56.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.375+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:39:56.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.375+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:39:56.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.376+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:39:56.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:56.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:56.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:57.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:57.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:57.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:57.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:58.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:58.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:58.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:58.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:39:59.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:59.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:39:59.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:39:59.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:00.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:00.637+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:00.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:00.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:00.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:00.693+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:01.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:01.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:01.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=18431 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f005d560).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:01.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=18431 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f005d560).stop 2021-03-09T10:40:01.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:01.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:01.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:01.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=CLOSED pgs=18431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:01.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:40:01.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:01.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:01.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:01.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:01.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:01.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:01.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:01.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:40:01.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:01.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:40:01.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:01.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030070 con 0x7f63d8003840 2021-03-09T10:40:01.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:01.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:01.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:01.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:01.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:01.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:01.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:40:01.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:01.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.373+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:01.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:01.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:01.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:01.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:40:01.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:40:01.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:01.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:01.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d3b0 con 0x7f63d8003840 2021-03-09T10:40:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026c20 con 0x7f63d8003840 2021-03-09T10:40:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:40:01.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.374+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:01.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.375+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:40:01.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.375+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:01.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.375+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 secure :-1 s=READY pgs=18435 cs=0 l=1 rev1=1 rx=0x7f63e002e360 tx=0x7f63fc0489c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:01.716 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:01.379+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51303 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:01.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f490 con 0x7f63d8003840 2021-03-09T10:40:01.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.379+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:01.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.379+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:01.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.379+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:01.718 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:01.719 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:01.719 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:01.719 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:01.720 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:01.720 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:01.720 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:01.720 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:01.721 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:01.721 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:01.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.379+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01f5f0 con 0x7f63d8003840 2021-03-09T10:40:01.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.379+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:01.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.379+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:40:01.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.381+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f490 con 0x7f63d8003840 2021-03-09T10:40:01.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:01.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:01.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:02.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:02.693+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:02.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:02.693+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:03.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:03.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:03.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:03.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:04.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:04.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:04.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:04.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:05.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:05.638+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:05.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:05.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:05.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:05.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:06.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:06.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:06.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 secure :-1 s=READY pgs=18435 cs=0 l=1 rev1=1 rx=0x7f63e002e360 tx=0x7f63fc0489c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:06.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 secure :-1 s=READY pgs=18435 cs=0 l=1 rev1=1 rx=0x7f63e002e360 tx=0x7f63fc0489c0).stop 2021-03-09T10:40:06.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:06.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:06.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:06.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 unknown :-1 s=CLOSED pgs=18435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:06.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:40:06.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:06.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:06.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:06.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:06.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:06.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:06.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:06.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d3c0 con 0x7f63d8003400 2021-03-09T10:40:06.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:06.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065ce0 con 0x7f63d8009130 2021-03-09T10:40:06.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:06.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:40:06.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:06.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:06.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:06.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:06.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.380+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:06.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:06.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:40:06.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:06.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:06.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:06.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:06.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:06.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:06.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:06.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:06.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:06.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:06.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:40:06.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:40:06.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.381+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:06.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:06.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:40:06.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:06.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:06.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d3b0 con 0x7f63d8005e00 2021-03-09T10:40:06.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:40:06.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:06.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=18449 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc058b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:06.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:40:06.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.382+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:06.397 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:06.386+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51342 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:06.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0475a0 con 0x7f63d8005e00 2021-03-09T10:40:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.387+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.387+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.387+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:06.398 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:06.399 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:06.400 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:06.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0475a0 con 0x7f63d8005e00 2021-03-09T10:40:06.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.387+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:06.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.387+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:40:06.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.388+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc066d80 con 0x7f63d8005e00 2021-03-09T10:40:06.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:06.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:06.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:07.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:07.694+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:07.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:07.694+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:08.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:08.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:08.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:08.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:09.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:09.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:09.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:09.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:10.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:10.639+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:10.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:10.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:10.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:10.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:10.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:10.695+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:11.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:11.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:11.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=18449 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc058b90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:11.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 secure :-1 s=READY pgs=18449 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc058b90).stop 2021-03-09T10:40:11.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:11.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:11.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:11.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001abe0 unknown :-1 s=CLOSED pgs=18449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:11.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:40:11.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:11.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:11.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:11.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:11.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:11.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:11.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:11.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079d30 con 0x7f63d8003400 2021-03-09T10:40:11.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:11.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8009130 2021-03-09T10:40:11.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:11.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068a10 con 0x7f63d8003840 2021-03-09T10:40:11.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:11.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:11.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.387+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.388+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.388+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:11.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.388+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.388+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:40:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.388+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.388+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.388+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:11.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.388+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.388+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:11.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:11.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:11.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:11.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:40:11.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:40:11.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0077100 con 0x7f63d8003840 2021-03-09T10:40:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:11.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065ec0 con 0x7f63d8003840 2021-03-09T10:40:11.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:40:11.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:11.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=18469 cs=0 l=1 rev1=1 rx=0x7f63f0006040 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:11.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:40:11.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.390+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:11.404 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:11.394+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51382 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:11.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.396+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:40:11.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.396+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:11.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.396+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:11.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.396+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:11.407 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:11.408 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:11.409 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:11.409 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:11.410 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:11.411 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:11.411 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.397+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:40:11.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.397+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.397+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:40:11.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.398+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0039330 con 0x7f63d8003840 2021-03-09T10:40:11.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:11.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:11.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:12.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:12.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:12.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:12.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:13.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:13.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:13.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:13.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:14.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:14.695+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:14.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:14.695+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:15.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:15.640+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:15.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:15.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:15.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:15.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:16.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:16.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:16.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=18469 cs=0 l=1 rev1=1 rx=0x7f63f0006040 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:16.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=18469 cs=0 l=1 rev1=1 rx=0x7f63f0006040 tx=0x7f63f004dd90).stop 2021-03-09T10:40:16.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:16.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:16.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:16.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=CLOSED pgs=18469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:40:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:16.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000fe20 con 0x7f63d8003400 2021-03-09T10:40:16.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bc20 con 0x7f63d8009130 2021-03-09T10:40:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:40:16.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:16.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:16.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:16.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:16.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:16.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.395+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:16.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.396+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:40:16.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.396+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:16.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.396+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:16.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.396+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:16.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.396+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:16.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.396+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:16.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:16.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:16.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:16.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:16.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:16.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:40:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:40:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0065ec0 con 0x7f63d8005e00 2021-03-09T10:40:16.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:16.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:16.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0077100 con 0x7f63d8005e00 2021-03-09T10:40:16.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:40:16.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.397+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:16.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.398+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008b40 secure :-1 s=READY pgs=18479 cs=0 l=1 rev1=1 rx=0x7f63f005ffd0 tx=0x7f63f004d110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:16.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.398+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:40:16.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.398+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:16.409 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:16.402+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51421 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:16.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:40:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.403+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.403+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.403+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:16.411 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:16.412 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:16.412 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:16.412 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:16.412 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:16.412 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:16.413 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:16.413 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:16.413 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:16.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.406+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:40:16.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.406+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:16.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.406+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:40:16.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.407+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8005e00 2021-03-09T10:40:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:16.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:16.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:17.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:17.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:17.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:17.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:18.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:18.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:18.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:18.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:19.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:19.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:19.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:19.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:20.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:20.641+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:20.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:20.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:20.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:20.696+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:21.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0008b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:21.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0008b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:21.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008b40 secure :-1 s=READY pgs=18479 cs=0 l=1 rev1=1 rx=0x7f63f005ffd0 tx=0x7f63f004d110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:21.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008b40 secure :-1 s=READY pgs=18479 cs=0 l=1 rev1=1 rx=0x7f63f005ffd0 tx=0x7f63f004d110).stop 2021-03-09T10:40:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0008b40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:21.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0008b40 unknown :-1 s=CLOSED pgs=18479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:21.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:40:21.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:21.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:21.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:21.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:21.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:21.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:21.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:21.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:40:21.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:21.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017a60 con 0x7f63d8009130 2021-03-09T10:40:21.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003840 2021-03-09T10:40:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:21.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:21.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:40:21.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:21.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:21.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:21.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.403+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:21.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:21.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:21.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:21.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:21.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:40:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:40:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:21.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:21.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006d3b0 con 0x7f63d8003840 2021-03-09T10:40:21.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:21.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:21.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0065ec0 con 0x7f63d8003840 2021-03-09T10:40:21.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:40:21.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:21.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=18490 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0089010).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:21.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:40:21.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.404+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:21.419 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:21.411+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51461 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:21.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.411+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002f1d0 con 0x7f63d8003840 2021-03-09T10:40:21.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.411+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:21.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.411+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:21.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.411+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:21.420 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:21.421 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:21.421 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:21.421 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:21.421 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:21.422 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:21.422 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:21.422 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:21.423 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:21.423 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:21.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.411+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002f1d0 con 0x7f63d8003840 2021-03-09T10:40:21.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.411+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:21.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.411+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:40:21.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0088b80 con 0x7f63d8003840 2021-03-09T10:40:21.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:21.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:21.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:22.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:22.696+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:22.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:22.696+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:23.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:23.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:23.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:23.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:24.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:24.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:24.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:24.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:25.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:25.642+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:25.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:25.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:25.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:25.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:26.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:26.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=18490 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0089010).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:26.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=18490 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0089010).stop 2021-03-09T10:40:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=CLOSED pgs=18490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:26.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:40:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:26.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066270 con 0x7f63d8003400 2021-03-09T10:40:26.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:26.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025cb0 con 0x7f63d8009130 2021-03-09T10:40:26.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:26.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030070 con 0x7f63d8005e00 2021-03-09T10:40:26.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:26.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:40:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:26.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.412+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:26.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:26.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:26.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:26.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:26.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:26.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:26.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:26.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:40:26.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:40:26.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:26.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:26.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T10:40:26.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:26.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:26.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006d3b0 con 0x7f63d8005e00 2021-03-09T10:40:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:40:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.413+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.414+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:40:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.414+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00092f0 secure :-1 s=READY pgs=18509 cs=0 l=1 rev1=1 rx=0x7f63f0036eb0 tx=0x7f63f007e8b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:26.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.414+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:26.426 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:26.418+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51500 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:26.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.420+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038e50 con 0x7f63d8005e00 2021-03-09T10:40:26.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.420+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:26.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.420+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:26.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.420+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:26.428 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:26.428 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:26.428 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:26.428 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:26.428 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:26.429 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:26.429 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:26.429 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:26.429 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:26.429 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:26.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.422+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0038e50 con 0x7f63d8005e00 2021-03-09T10:40:26.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.422+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:26.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.422+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:40:26.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.423+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0088b80 con 0x7f63d8005e00 2021-03-09T10:40:26.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:26.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:26.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:27.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:27.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:27.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:28.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:28.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:28.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:28.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:29.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:29.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:29.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:29.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:30.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:30.643+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:30.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:30.698+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:30.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:30.698+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:30.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:30.698+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:31.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:31.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:31.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00092f0 secure :-1 s=READY pgs=18509 cs=0 l=1 rev1=1 rx=0x7f63f0036eb0 tx=0x7f63f007e8b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:31.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00092f0 secure :-1 s=READY pgs=18509 cs=0 l=1 rev1=1 rx=0x7f63f0036eb0 tx=0x7f63f007e8b0).stop 2021-03-09T10:40:31.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:31.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:31.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00092f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:31.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00092f0 unknown :-1 s=CLOSED pgs=18509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:31.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:40:31.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:31.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:31.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:31.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:31.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:31.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:31.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:31.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:40:31.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:31.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006aaf0 con 0x7f63d8009130 2021-03-09T10:40:31.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:31.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e070 con 0x7f63d8003840 2021-03-09T10:40:31.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:31.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:31.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:31.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:31.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.419+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:31.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.420+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:31.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.420+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:40:31.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.420+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:31.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.420+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:31.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.420+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:31.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.420+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:31.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.420+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:31.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:31.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:31.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:31.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:31.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:31.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:40:31.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:40:31.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:31.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:31.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0077100 con 0x7f63d8003840 2021-03-09T10:40:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.421+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8003840 2021-03-09T10:40:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.422+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:40:31.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.422+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.422+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 secure :-1 s=READY pgs=18523 cs=0 l=1 rev1=1 rx=0x7f63f007b000 tx=0x7f63f00554f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.422+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:40:31.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.422+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:31.433 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:31.426+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51538 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:31.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f4c0 con 0x7f63d8003840 2021-03-09T10:40:31.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.426+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:31.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.426+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:31.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.426+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:31.435 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:31.435 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:31.436 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:31.437 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:31.437 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:31.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8003840 2021-03-09T10:40:31.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.430+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:31.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.430+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:40:31.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.430+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8003840 2021-03-09T10:40:31.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.697+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:31.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:31.697+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:32.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:32.698+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:32.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:32.698+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:33.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:33.698+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:33.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:33.698+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:34.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:34.698+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:34.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:34.698+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:35.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:35.644+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:35.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:35.698+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:35.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:35.698+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:36.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00235a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:36.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00235a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:36.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 secure :-1 s=READY pgs=18523 cs=0 l=1 rev1=1 rx=0x7f63f007b000 tx=0x7f63f00554f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:36.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 secure :-1 s=READY pgs=18523 cs=0 l=1 rev1=1 rx=0x7f63f007b000 tx=0x7f63f00554f0).stop 2021-03-09T10:40:36.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:36.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:36.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00235a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:36.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 unknown :-1 s=CLOSED pgs=18523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:36.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:40:36.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:36.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:36.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:36.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:36.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:36.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:36.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:36.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002de20 con 0x7f63d8003400 2021-03-09T10:40:36.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:36.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:40:36.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:36.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025cb0 con 0x7f63d8005e00 2021-03-09T10:40:36.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:36.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:36.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:36.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:36.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.426+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:36.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.427+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:36.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.427+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:40:36.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.427+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:36.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.427+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:36.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.428+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:36.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.428+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:36.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.428+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:36.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:36.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:36.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:40:36.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:40:36.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:36.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:36.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00216e0 con 0x7f63d8005e00 2021-03-09T10:40:36.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:36.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:36.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0077100 con 0x7f63d8005e00 2021-03-09T10:40:36.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:40:36.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.429+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:36.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.430+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000fc20 secure :-1 s=READY pgs=18541 cs=0 l=1 rev1=1 rx=0x7f63f007b000 tx=0x7f63f0003500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:36.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.430+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:40:36.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.430+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:36.443 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:36.433+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51579 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:36.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007c350 con 0x7f63d8005e00 2021-03-09T10:40:36.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.434+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:36.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.434+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:36.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.434+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:36.445 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:36.445 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:36.445 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:36.446 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:36.446 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:36.446 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:36.447 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:36.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.436+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0042910 con 0x7f63d8005e00 2021-03-09T10:40:36.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.436+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:36.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.436+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:40:36.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:40:36.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.698+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:36.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:36.698+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:37.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:37.699+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:37.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:37.699+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:38.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:38.699+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:38.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:38.699+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:39.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:39.699+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:39.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:39.699+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:40.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:40.645+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:40.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:40.699+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:40.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:40.699+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:40.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:40.699+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:41.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:41.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:41.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000fc20 secure :-1 s=READY pgs=18541 cs=0 l=1 rev1=1 rx=0x7f63f007b000 tx=0x7f63f0003500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:41.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000fc20 secure :-1 s=READY pgs=18541 cs=0 l=1 rev1=1 rx=0x7f63f007b000 tx=0x7f63f0003500).stop 2021-03-09T10:40:41.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:41.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:41.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e000fc20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:41.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e000fc20 unknown :-1 s=CLOSED pgs=18541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:40:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:41.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:41.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030070 con 0x7f63d8003400 2021-03-09T10:40:41.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:41.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:40:41.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:41.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b4c0 con 0x7f63d8003840 2021-03-09T10:40:41.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:41.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:41.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:41.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:41.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:40:41.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:41.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.434+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:41.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.435+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:41.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.435+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:41.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.435+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:41.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:41.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:41.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:41.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:41.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:41.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:40:41.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:40:41.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:41.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:41.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8003840 2021-03-09T10:40:41.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:41.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:41.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:40:41.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:40:41.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.436+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:41.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.437+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 secure :-1 s=READY pgs=18551 cs=0 l=1 rev1=1 rx=0x7f63e002a470 tx=0x7f63fc048a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:41.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.437+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:40:41.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.437+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:41.450 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:41.442+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51619 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:41.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f63d8003840 2021-03-09T10:40:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.443+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.443+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.443+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:41.452 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:41.453 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:41.453 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:41.453 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:41.453 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:41.454 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:41.454 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:41.454 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:41.455 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:41.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.444+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f63d8003840 2021-03-09T10:40:41.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.444+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:41.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.444+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:40:41.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.445+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:40:41.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.698+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:41.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:41.698+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:42.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:42.699+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:42.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:42.699+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:43.699+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:43.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:43.699+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:44.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:44.700+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:44.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:44.700+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:45.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:45.646+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:45.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:45.700+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:45.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:45.700+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00235a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00235a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 secure :-1 s=READY pgs=18551 cs=0 l=1 rev1=1 rx=0x7f63e002a470 tx=0x7f63fc048a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:46.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 secure :-1 s=READY pgs=18551 cs=0 l=1 rev1=1 rx=0x7f63e002a470 tx=0x7f63fc048a30).stop 2021-03-09T10:40:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00235a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:46.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00235a0 unknown :-1 s=CLOSED pgs=18551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:40:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:46.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068a10 con 0x7f63d8003400 2021-03-09T10:40:46.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:46.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ed90 con 0x7f63d8009130 2021-03-09T10:40:46.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:46.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:40:46.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:46.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.442+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:46.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.443+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:46.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.443+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:46.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.443+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:46.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.444+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:46.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.444+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:40:46.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.444+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:46.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.444+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:46.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.444+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:46.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.444+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:46.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.444+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:46.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:46.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:46.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:46.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:46.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:40:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:40:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:46.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b5c0 con 0x7f63d8005e00 2021-03-09T10:40:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T10:40:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:40:46.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.445+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:46.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.446+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18568 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc05d850).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:46.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.446+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:40:46.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.446+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:46.457 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:46.449+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51658 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:46.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0475e0 con 0x7f63d8005e00 2021-03-09T10:40:46.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.450+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:46.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.450+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:46.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.450+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:46.459 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:46.459 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:46.460 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:46.460 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:46.460 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:46.460 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:46.460 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:46.461 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:46.461 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:46.461 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:46.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.454+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01fba0 con 0x7f63d8005e00 2021-03-09T10:40:46.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.454+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:46.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.454+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8005e00 2021-03-09T10:40:46.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.454+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:40:46.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.700+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:46.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:46.700+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:47.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:47.700+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:47.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:47.700+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:48.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:48.700+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:48.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:48.700+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:49.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:49.700+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:49.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:49.700+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:50.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:50.647+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:50.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:50.700+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:50.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:50.700+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:50.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:50.700+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:51.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.450+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:40:51.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.450+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.450+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18568 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc05d850).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.450+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18568 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc05d850).stop 2021-03-09T10:40:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:51.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=18568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:40:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:51.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:51.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bb40 con 0x7f63d8003400 2021-03-09T10:40:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d2c0 con 0x7f63d8009130 2021-03-09T10:40:51.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030070 con 0x7f63d8003840 2021-03-09T10:40:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:51.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:40:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:51.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:51.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.451+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:51.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:51.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:51.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:51.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:51.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:51.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:40:51.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:40:51.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:51.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:51.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00266a0 con 0x7f63d8003840 2021-03-09T10:40:51.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:51.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:51.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00216e0 con 0x7f63d8003840 2021-03-09T10:40:51.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:40:51.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.452+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:51.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.453+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c2b0 secure :-1 s=READY pgs=18576 cs=0 l=1 rev1=1 rx=0x7f63f00428a0 tx=0x7f63f007e920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:51.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.453+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:40:51.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.453+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:51.466 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:51.457+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51696 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:51.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:40:51.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.458+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:51.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.458+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:51.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.458+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:51.468 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:51.468 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:51.468 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:51.468 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:51.469 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:51.469 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:51.469 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:51.469 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:51.470 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:51.470 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:51.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028e90 con 0x7f63d8003840 2021-03-09T10:40:51.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.461+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:51.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.461+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:40:51.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.462+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63d8003840 2021-03-09T10:40:51.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.700+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:51.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:51.700+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:52.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:52.701+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:52.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:52.701+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:53.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:53.701+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:53.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:53.701+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:54.701+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:54.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:54.701+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:55.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:55.648+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:40:55.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:55.701+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:55.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:55.701+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:40:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:40:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c2b0 secure :-1 s=READY pgs=18576 cs=0 l=1 rev1=1 rx=0x7f63f00428a0 tx=0x7f63f007e920).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:40:56.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c2b0 secure :-1 s=READY pgs=18576 cs=0 l=1 rev1=1 rx=0x7f63f00428a0 tx=0x7f63f007e920).stop 2021-03-09T10:40:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:40:56.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002c2b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:40:56.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002c2b0 unknown :-1 s=CLOSED pgs=18576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:40:56.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:40:56.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:40:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:40:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:40:56.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:40:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:40:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:56.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bb40 con 0x7f63d8009130 2021-03-09T10:40:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:40:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e070 con 0x7f63d8005e00 2021-03-09T10:40:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:40:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:40:56.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.458+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:56.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.459+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:40:56.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.459+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:40:56.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.459+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:40:56.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.459+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:40:56.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.459+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:40:56.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.459+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:40:56.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:40:56.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:40:56.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:40:56.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:40:56.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:56.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:56.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:40:56.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:40:56.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:40:56.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:40:56.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:40:56.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:56.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8005e00 2021-03-09T10:40:56.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:40:56.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:40:56.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b5c0 con 0x7f63d8005e00 2021-03-09T10:40:56.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:40:56.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.460+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:56.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.462+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:40:56.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.462+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18596 cs=0 l=1 rev1=1 rx=0x7f63e006e110 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:40:56.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.462+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:40:56.475 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:40:56.465+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51737 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:40:56.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.466+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63d8005e00 2021-03-09T10:40:56.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.466+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:40:56.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.466+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:40:56.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.466+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:40:56.477 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:40:56.477 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:40:56.477 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:56.477 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:40:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:40:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:40:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:40:56.478 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:40:56.479 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:40:56.479 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:40:56.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.466+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0475e0 con 0x7f63d8005e00 2021-03-09T10:40:56.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.466+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:40:56.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.466+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:40:56.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:40:56.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.701+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:56.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:56.701+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:57.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:57.701+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:57.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:57.701+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:58.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:58.702+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:58.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:58.702+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:40:59.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:59.702+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:40:59.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:40:59.702+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:00.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:00.649+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:00.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:00.702+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:00.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:00.702+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:00.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:00.702+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:01.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:01.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:01.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18596 cs=0 l=1 rev1=1 rx=0x7f63e006e110 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:01.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18596 cs=0 l=1 rev1=1 rx=0x7f63e006e110 tx=0x7f63fc04a6d0).stop 2021-03-09T10:41:01.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:01.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:01.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:01.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=18596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:01.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:41:01.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:01.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:01.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:01.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:01.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:01.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:01.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:01.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T10:41:01.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:01.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:41:01.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:01.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003840 2021-03-09T10:41:01.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:01.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:01.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:01.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:01.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.467+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:01.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.468+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:01.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.468+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:41:01.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.468+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:01.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.468+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:01.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.468+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:01.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.468+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:01.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.468+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:01.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:41:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:41:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:01.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:41:01.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:01.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:01.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00266a0 con 0x7f63d8003840 2021-03-09T10:41:01.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:41:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.469+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.470+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006c5b0 secure :-1 s=READY pgs=18603 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0031f30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.470+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:41:01.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.470+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:01.482 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:01.475+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51777 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:01.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.475+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:41:01.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.475+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:01.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.475+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:01.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.475+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:01.484 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:01.485 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:01.486 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:41:01.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.476+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.476+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:41:01.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.477+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63d8003840 2021-03-09T10:41:01.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.702+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:01.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:01.702+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:02.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:02.702+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:02.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:02.702+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:03.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:03.702+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:03.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:03.702+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:04.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:04.702+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:04.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:04.702+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:05.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:05.650+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:05.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:05.702+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:05.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:05.703+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:06.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:06.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.475+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:06.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.475+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006c5b0 secure :-1 s=READY pgs=18603 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0031f30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:06.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006c5b0 secure :-1 s=READY pgs=18603 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f0031f30).stop 2021-03-09T10:41:06.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:06.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:06.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006c5b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:06.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006c5b0 unknown :-1 s=CLOSED pgs=18603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:06.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:41:06.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:06.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:06.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:06.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006af10 con 0x7f63d8003400 2021-03-09T10:41:06.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:06.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030070 con 0x7f63d8009130 2021-03-09T10:41:06.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:06.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:41:06.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:06.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:06.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.476+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:06.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:06.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:41:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:06.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:06.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:06.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:06.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:06.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:06.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:06.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:06.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.477+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:06.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:41:06.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:41:06.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:06.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:06.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0002cf0 con 0x7f63d8005e00 2021-03-09T10:41:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:41:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:41:06.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:06.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:41:06.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006caa0 secure :-1 s=READY pgs=18615 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f0028f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:06.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.478+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:06.491 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:06.483+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51816 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:06.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.483+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8005e00 2021-03-09T10:41:06.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.483+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:06.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.483+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.483+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:06.493 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:06.494 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:06.494 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:06.494 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:06.495 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:06.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.483+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:41:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.483+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.483+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:41:06.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.484+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028b20 con 0x7f63d8005e00 2021-03-09T10:41:06.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.703+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:06.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:06.703+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:07.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:07.703+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:07.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:07.703+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:08.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:08.703+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:08.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:08.703+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:09.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:09.703+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:09.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:09.703+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:10.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:10.651+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:10.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:10.703+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:10.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:10.703+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:10.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:10.703+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006caa0 secure :-1 s=READY pgs=18615 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f0028f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:11.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006caa0 secure :-1 s=READY pgs=18615 cs=0 l=1 rev1=1 rx=0x7f63f0013800 tx=0x7f63f0028f40).stop 2021-03-09T10:41:11.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:11.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:11.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006caa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:11.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006caa0 unknown :-1 s=CLOSED pgs=18615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:41:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:11.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:11.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068c10 con 0x7f63d8003400 2021-03-09T10:41:11.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:11.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8009130 2021-03-09T10:41:11.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:11.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d2c0 con 0x7f63d8003840 2021-03-09T10:41:11.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:11.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:11.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.483+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:11.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.484+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:11.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.484+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:11.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.484+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:11.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.484+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:41:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.484+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.484+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.484+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:11.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.484+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.484+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:11.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:11.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:11.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:11.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:41:11.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:41:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066260 con 0x7f63d8003840 2021-03-09T10:41:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:11.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:41:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:41:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=18635 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc071180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:11.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:41:11.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.486+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:11.498 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:11.490+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51857 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:11.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0128a0 con 0x7f63d8003840 2021-03-09T10:41:11.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.490+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:11.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.490+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:11.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.490+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:11.500 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:11.500 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:11.500 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:11.500 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:11.501 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:11.502 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.493+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0128a0 con 0x7f63d8003840 2021-03-09T10:41:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.493+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:11.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8003840 2021-03-09T10:41:11.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.494+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:41:11.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.703+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:11.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:11.703+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:12.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:12.703+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:12.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:12.703+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:13.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:13.704+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:13.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:13.704+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:14.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:14.704+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:14.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:14.704+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:15.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:15.652+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:15.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:15.704+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:15.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:15.704+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:16.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:16.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:16.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=18635 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc071180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:16.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=18635 cs=0 l=1 rev1=1 rx=0x7f63fc006e50 tx=0x7f63fc071180).stop 2021-03-09T10:41:16.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:16.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:16.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:16.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=18635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:16.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:41:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:16.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:16.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:16.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:16.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T10:41:16.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:16.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006af10 con 0x7f63d8009130 2021-03-09T10:41:16.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:16.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8005e00 2021-03-09T10:41:16.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:16.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:16.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.491+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:16.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.492+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:16.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.492+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:16.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.492+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:16.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.492+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:41:16.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.492+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:16.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.492+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:16.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.493+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:16.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.493+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:16.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.493+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:16.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:16.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:16.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:16.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:16.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:16.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:41:16.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:41:16.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:16.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:16.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T10:41:16.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:16.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:16.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066260 con 0x7f63d8005e00 2021-03-09T10:41:16.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:41:16.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:16.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021990 secure :-1 s=READY pgs=18654 cs=0 l=1 rev1=1 rx=0x7f63e001d760 tx=0x7f63fc066ba0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:16.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:41:16.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.495+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:16.505 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:16.498+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51895 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:16.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f63d8005e00 2021-03-09T10:41:16.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.499+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:16.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.499+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:16.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.499+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:16.507 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:16.507 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:16.507 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:16.507 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:16.507 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:16.508 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:16.508 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:16.508 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:16.508 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:16.508 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:16.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f63d8005e00 2021-03-09T10:41:16.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.502+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:16.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.502+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:41:16.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.504+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc009520 con 0x7f63d8005e00 2021-03-09T10:41:16.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.704+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:16.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:16.704+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:17.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:17.704+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:17.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:17.704+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:18.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:18.704+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:18.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:18.704+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:19.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:19.704+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:19.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:19.704+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:20.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:20.653+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:20.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:20.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:20.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:20.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:20.705+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:21.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:21.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:21.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021990 secure :-1 s=READY pgs=18654 cs=0 l=1 rev1=1 rx=0x7f63e001d760 tx=0x7f63fc066ba0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:21.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021990 secure :-1 s=READY pgs=18654 cs=0 l=1 rev1=1 rx=0x7f63e001d760 tx=0x7f63fc066ba0).stop 2021-03-09T10:41:21.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:21.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:21.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0021990 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:21.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0021990 unknown :-1 s=CLOSED pgs=18654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:21.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:41:21.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:21.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:21.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:21.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:21.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:21.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00669a0 con 0x7f63d8003400 2021-03-09T10:41:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:41:21.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:21.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ed90 con 0x7f63d8003840 2021-03-09T10:41:21.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:21.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:21.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:21.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:21.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.499+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:21.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.500+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:21.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.500+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:41:21.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.500+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:21.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.500+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:21.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.500+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:21.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.500+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:21.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.500+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:21.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:21.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:21.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:21.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:21.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:21.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:41:21.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:41:21.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:21.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:21.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:41:21.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:21.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:21.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0002cf0 con 0x7f63d8003840 2021-03-09T10:41:21.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:41:21.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.501+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:21.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.502+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ce30 secure :-1 s=READY pgs=18662 cs=0 l=1 rev1=1 rx=0x7f63f0013470 tx=0x7f63f004dab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:21.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.502+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:41:21.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.502+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:21.512 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:21.506+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51936 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:21.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f650 con 0x7f63d8003840 2021-03-09T10:41:21.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.506+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:21.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.506+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:21.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.506+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:21.514 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:21.514 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:21.515 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:21.515 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:21.515 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:21.515 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:21.515 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:21.516 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:21.516 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:21.516 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:21.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.508+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007f650 con 0x7f63d8003840 2021-03-09T10:41:21.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.508+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:21.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.508+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:41:21.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.510+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8003840 2021-03-09T10:41:21.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:21.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:21.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:22.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:22.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:22.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:22.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:23.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:23.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:23.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:23.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:24.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:24.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:24.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:24.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:25.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:25.654+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:25.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:25.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:25.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:25.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:26.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.505+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:26.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.505+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:26.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.505+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ce30 secure :-1 s=READY pgs=18662 cs=0 l=1 rev1=1 rx=0x7f63f0013470 tx=0x7f63f004dab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:26.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ce30 secure :-1 s=READY pgs=18662 cs=0 l=1 rev1=1 rx=0x7f63f0013470 tx=0x7f63f004dab0).stop 2021-03-09T10:41:26.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:26.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:26.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006ce30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:26.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006ce30 unknown :-1 s=CLOSED pgs=18662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:26.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:41:26.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00298f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:26.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:26.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:26.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:26.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:26.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:26.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:26.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025cb0 con 0x7f63d8003400 2021-03-09T10:41:26.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:26.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016fb0 con 0x7f63d8009130 2021-03-09T10:41:26.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:26.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:41:26.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:26.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:26.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.506+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:26.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.507+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:26.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.507+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:26.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.507+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00298f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:26.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.507+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:41:26.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.507+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:26.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.507+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:26.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:26.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:26.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:26.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:26.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:26.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:26.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:26.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:26.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:41:26.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:41:26.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:26.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:26.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068460 con 0x7f63d8005e00 2021-03-09T10:41:26.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:26.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:26.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8005e00 2021-03-09T10:41:26.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:41:26.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.508+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:26.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.509+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:41:26.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.509+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:26.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.509+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00298f0 secure :-1 s=READY pgs=18677 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0027e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:26.524 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:26.513+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 51974 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:26.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:41:26.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.514+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:26.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.514+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:26.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.514+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:26.525 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:26.526 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:26.526 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:26.526 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:26.526 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:26.527 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:26.527 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:26.527 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:26.527 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:26.527 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:26.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00481f0 con 0x7f63d8005e00 2021-03-09T10:41:26.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.514+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:26.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.514+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:41:26.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.515+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028b20 con 0x7f63d8005e00 2021-03-09T10:41:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.704+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:26.704+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:27.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:27.704+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:27.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:27.704+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:28.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:28.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:28.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:28.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:29.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:29.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:29.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:30.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:30.654+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:30.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:30.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:30.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:30.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:30.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:30.705+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:31.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.513+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00298f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:31.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.513+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00298f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:31.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.513+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00298f0 secure :-1 s=READY pgs=18677 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0027e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:31.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.513+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00298f0 secure :-1 s=READY pgs=18677 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0027e00).stop 2021-03-09T10:41:31.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:31.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:31.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00298f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:31.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00298f0 unknown :-1 s=CLOSED pgs=18677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:31.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:41:31.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:31.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:31.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:31.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:31.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:31.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:31.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:31.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006af10 con 0x7f63d8003400 2021-03-09T10:41:31.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:31.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077390 con 0x7f63d8009130 2021-03-09T10:41:31.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:31.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a730 con 0x7f63d8003840 2021-03-09T10:41:31.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:31.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:31.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:31.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:31.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:31.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:31.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:41:31.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:31.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.514+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:31.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.516+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:31.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.516+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:31.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.516+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:31.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:31.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:31.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:41:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:41:31.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013150 con 0x7f63d8003840 2021-03-09T10:41:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:31.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068460 con 0x7f63d8003840 2021-03-09T10:41:31.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:41:31.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:31.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=18689 cs=0 l=1 rev1=1 rx=0x7f63f0027d90 tx=0x7f63f0027c80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:31.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:41:31.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.517+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:31.529 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:31.521+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52015 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:31.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63d8003840 2021-03-09T10:41:31.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.522+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:31.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.522+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.522+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:31.531 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:31.532 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:31.532 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:31.532 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:31.532 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:31.533 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:31.533 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:31.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040bc0 con 0x7f63d8003840 2021-03-09T10:41:31.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.524+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:31.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.524+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:41:31.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.525+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:41:31.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:31.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:31.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:32.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:32.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:32.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:32.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:33.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:33.705+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:33.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:33.705+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:34.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:34.706+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:34.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:34.706+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:35.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:35.655+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:35.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:35.706+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:35.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:35.706+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:36.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:36.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:36.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=18689 cs=0 l=1 rev1=1 rx=0x7f63f0027d90 tx=0x7f63f0027c80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:36.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=18689 cs=0 l=1 rev1=1 rx=0x7f63f0027d90 tx=0x7f63f0027c80).stop 2021-03-09T10:41:36.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:36.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:36.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:36.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=18689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:36.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:41:36.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:36.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:36.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:36.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:36.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:36.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:36.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:36.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00669a0 con 0x7f63d8003400 2021-03-09T10:41:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:41:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8005e00 2021-03-09T10:41:36.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.521+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.522+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:36.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.522+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:36.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.522+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:36.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.522+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:41:36.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.522+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.522+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.523+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.523+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.523+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:36.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.523+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:36.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:36.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:36.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:36.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:36.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:41:36.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:41:36.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:36.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:36.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027fb0 con 0x7f63d8005e00 2021-03-09T10:41:36.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:36.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:36.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013150 con 0x7f63d8005e00 2021-03-09T10:41:36.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:41:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.524+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.525+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:41:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.525+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006a250 secure :-1 s=READY pgs=18705 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f002d290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:36.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.525+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:36.537 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:36.529+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52053 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:36.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004c260 con 0x7f63d8005e00 2021-03-09T10:41:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.529+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.529+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.529+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:36.538 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:36.539 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:36.539 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:36.539 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:36.539 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:36.540 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:36.540 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:36.540 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:36.540 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:36.540 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:36.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.533+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e660 con 0x7f63d8005e00 2021-03-09T10:41:36.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.533+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:36.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.533+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:41:36.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.534+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8005e00 2021-03-09T10:41:36.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.706+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:36.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:36.706+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:37.706+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:37.706+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:38.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:38.706+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:38.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:38.706+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:39.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:39.706+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:39.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:39.706+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:40.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:40.656+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:40.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:40.706+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:40.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:40.706+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:40.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:40.706+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:41.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:41.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:41.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006a250 secure :-1 s=READY pgs=18705 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f002d290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:41.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006a250 secure :-1 s=READY pgs=18705 cs=0 l=1 rev1=1 rx=0x7f63f0023190 tx=0x7f63f002d290).stop 2021-03-09T10:41:41.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e006a250 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e006a250 unknown :-1 s=CLOSED pgs=18705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:41:41.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:41.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:41.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:41.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:41.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006af10 con 0x7f63d8003400 2021-03-09T10:41:41.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:41.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020b10 con 0x7f63d8009130 2021-03-09T10:41:41.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:41.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:41:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.529+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:41.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:41.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:41:41.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:41.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:41.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:41.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:41.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.530+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:41.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:41.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:41.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:41.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:41.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:41.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:41:41.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:41:41.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:41.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:41.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079e90 con 0x7f63d8003840 2021-03-09T10:41:41.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:41.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:41.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8003840 2021-03-09T10:41:41.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:41:41.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:41.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067b80 secure :-1 s=READY pgs=18713 cs=0 l=1 rev1=1 rx=0x7f63fc066eb0 tx=0x7f63fc072ce0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:41.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:41:41.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.531+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:41.546 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:41.537+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52104 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:41.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066f20 con 0x7f63d8003840 2021-03-09T10:41:41.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.537+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:41.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.537+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.537+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:41.548 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:41.549 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:41.549 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:41.549 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:41.549 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:41.549 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:41.550 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:41.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.537+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc048960 con 0x7f63d8003840 2021-03-09T10:41:41.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.537+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:41.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.537+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:41:41.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058b90 con 0x7f63d8003840 2021-03-09T10:41:41.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.706+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:41.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:41.706+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:42.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:42.707+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:42.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:42.707+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:43.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:43.707+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:43.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:43.707+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:44.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:44.707+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:44.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:44.707+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:45.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:45.657+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:45.707+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:45.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:45.707+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.537+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:46.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.537+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.537+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067b80 secure :-1 s=READY pgs=18713 cs=0 l=1 rev1=1 rx=0x7f63fc066eb0 tx=0x7f63fc072ce0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067b80 secure :-1 s=READY pgs=18713 cs=0 l=1 rev1=1 rx=0x7f63fc066eb0 tx=0x7f63fc072ce0).stop 2021-03-09T10:41:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:46.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:46.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067b80 unknown :-1 s=CLOSED pgs=18713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:46.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:41:46.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:46.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:46.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:46.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:46.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:46.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:46.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T10:41:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:41:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:46.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006af10 con 0x7f63d8005e00 2021-03-09T10:41:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:46.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:46.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:41:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.538+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:46.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.539+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.539+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.539+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.539+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.539+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:46.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.539+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:41:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:41:46.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:46.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:46.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8005e00 2021-03-09T10:41:46.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:46.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:46.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079e90 con 0x7f63d8005e00 2021-03-09T10:41:46.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:41:46.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.540+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:46.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.541+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:41:46.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.541+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18731 cs=0 l=1 rev1=1 rx=0x7f63e0012d50 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:46.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.541+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:46.552 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:46.546+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52142 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:46.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc048960 con 0x7f63d8005e00 2021-03-09T10:41:46.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.547+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:46.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.547+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:46.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.547+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:46.554 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:46.554 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:46.554 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:46.554 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:46.554 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:46.555 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:46.555 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:46.555 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:46.555 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:46.555 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:46.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.550+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc048960 con 0x7f63d8005e00 2021-03-09T10:41:46.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.550+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:46.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.550+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:41:46.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.551+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc008580 con 0x7f63d8005e00 2021-03-09T10:41:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.707+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:46.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:46.707+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:47.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:47.707+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:47.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:47.707+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:48.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:48.707+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:48.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:48.707+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:49.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:49.708+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:49.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:49.708+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:50.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:50.658+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:50.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:50.708+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:50.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:50.708+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:50.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:50.708+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:51.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:51.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:51.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18731 cs=0 l=1 rev1=1 rx=0x7f63e0012d50 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:51.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 secure :-1 s=READY pgs=18731 cs=0 l=1 rev1=1 rx=0x7f63e0012d50 tx=0x7f63fc009a00).stop 2021-03-09T10:41:51.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:51.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:51.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:51.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=18731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:51.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:41:51.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:51.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002fd20 con 0x7f63d8003400 2021-03-09T10:41:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d1d0 con 0x7f63d8009130 2021-03-09T10:41:51.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:51.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:41:51.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:51.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:51.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:51.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:51.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:51.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:51.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:41:51.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:51.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:51.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:51.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:51.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.547+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:51.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:51.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:51.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:51.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:51.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:51.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:41:51.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:41:51.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:51.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:51.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006a440 con 0x7f63d8003840 2021-03-09T10:41:51.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:51.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:51.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8003840 2021-03-09T10:41:51.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:41:51.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:51.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:41:51.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=18739 cs=0 l=1 rev1=1 rx=0x7f63e001ba00 tx=0x7f63fc05d030).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:51.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.548+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:51.561 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:51.555+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52183 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:51.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023bc0 con 0x7f63d8003840 2021-03-09T10:41:51.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.555+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:51.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.555+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:51.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.555+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:51.563 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:51.563 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:51.564 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:51.564 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:51.564 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:51.564 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:51.565 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:51.565 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:51.565 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:51.565 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:51.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023bc0 con 0x7f63d8003840 2021-03-09T10:41:51.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.555+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:51.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.555+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:41:51.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc008580 con 0x7f63d8003840 2021-03-09T10:41:51.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.708+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:51.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:51.708+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:52.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:52.708+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:52.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:52.708+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:53.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:53.708+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:53.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:53.708+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:54.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:54.708+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:54.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:54.708+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:55.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:55.659+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:41:55.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:55.708+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:55.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:55.708+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:56.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:41:56.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:56.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=18739 cs=0 l=1 rev1=1 rx=0x7f63e001ba00 tx=0x7f63fc05d030).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:56.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=18739 cs=0 l=1 rev1=1 rx=0x7f63e001ba00 tx=0x7f63fc05d030).stop 2021-03-09T10:41:56.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:56.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:56.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:56.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=CLOSED pgs=18739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:56.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:41:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:56.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T10:41:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:56.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:41:56.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:56.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e990 con 0x7f63d8005e00 2021-03-09T10:41:56.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.555+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.556+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.556+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.556+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.556+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:56.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:41:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:56.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:56.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:56.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:56.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:56.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:56.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:56.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:56.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:56.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:41:56.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:41:56.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:56.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:56.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017980 con 0x7f63d8005e00 2021-03-09T10:41:56.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:56.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:56.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.557+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027fb0 con 0x7f63d8005e00 2021-03-09T10:41:56.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.558+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:41:56.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.558+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:56.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.558+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:41:56.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.558+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:41:56.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.558+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0x7f63f0006c40 tx=0x7f63f007b530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:41:56.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.558+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:56.617 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:41:56.562+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52221 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:41:56.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.709+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:56.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.709+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:56.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.709+0000 7f63ecff9700 10 monclient: renew subs? -- yes 2021-03-09T10:41:56.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.709+0000 7f63ecff9700 10 monclient: _renew_subs - empty 2021-03-09T10:41:56.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f6405d52700 10 monclient: get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:41:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f6405d52700 0 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-03-09T10:41:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:41:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:41:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:41:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:41:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:41:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:41:56.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:41:56.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:41:56.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:56.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006af10 con 0x7f63d8003400 2021-03-09T10:41:56.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:56.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8009130 2021-03-09T10:41:56.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:41:56.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:41:56.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:41:56.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:41:56.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:56.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:41:56.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:41:56.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:41:56.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:41:56.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:41:56.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:41:56.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:41:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:41:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:41:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:41:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:41:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:41:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:41:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:41:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:41:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013150 con 0x7f63d8003840 2021-03-09T10:41:56.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:41:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:41:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017980 con 0x7f63d8003840 2021-03-09T10:41:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:41:56.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:56.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=18767 cs=0 l=1 rev1=1 rx=0x7f63f0029ee0 tx=0x7f63f0055470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:41:56.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:41:56.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:41:56.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:41:56.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:41:56.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:41:56.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:41:56.777 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:41:56.777 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:41:56.778 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:56.778 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:41:56.778 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:41:56.778 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:41:56.778 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:41:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:41:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:41:56.779 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:41:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:41:56.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:41:56.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.760+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:41:56.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:56.761+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0073340 con 0x7f63d8003840 2021-03-09T10:41:57.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:57.709+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:57.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:57.709+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:58.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:58.709+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:58.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:58.709+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:41:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:59.709+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:41:59.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:41:59.709+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:00.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:00.660+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:00.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:00.709+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:00.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:00.709+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:00.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:00.709+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:01.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:01.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:01.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=18767 cs=0 l=1 rev1=1 rx=0x7f63f0029ee0 tx=0x7f63f0055470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:01.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 secure :-1 s=READY pgs=18767 cs=0 l=1 rev1=1 rx=0x7f63f0029ee0 tx=0x7f63f0055470).stop 2021-03-09T10:42:01.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:01.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:01.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005d7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:01.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005d7d0 unknown :-1 s=CLOSED pgs=18767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:01.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:42:01.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:01.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:01.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:01.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:01.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:01.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.563+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:01.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:01.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e990 con 0x7f63d8003400 2021-03-09T10:42:01.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:01.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:42:01.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:01.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:42:01.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:01.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:01.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:01.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:01.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:01.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:01.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.564+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:42:01.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:01.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:01.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:01.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:01.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:01.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:42:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:42:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:01.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027fb0 con 0x7f63d8005e00 2021-03-09T10:42:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013150 con 0x7f63d8005e00 2021-03-09T10:42:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:42:01.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.565+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:01.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.566+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18773 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0055340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:01.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.566+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:42:01.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.566+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:01.579 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:01.570+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52260 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:01.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.570+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028eb0 con 0x7f63d8005e00 2021-03-09T10:42:01.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.571+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:01.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.571+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:01.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.571+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:01.581 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:01.582 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:01.582 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:01.582 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:01.582 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:01.582 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:01.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0028eb0 con 0x7f63d8005e00 2021-03-09T10:42:01.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.573+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:01.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.573+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:42:01.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.575+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8005e00 2021-03-09T10:42:01.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.709+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:01.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:01.709+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:02.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:02.709+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:02.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:02.709+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:03.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:03.710+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:03.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:03.710+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:04.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:04.709+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:04.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:04.709+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:05.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:05.661+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:05.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:05.710+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:05.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:05.710+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:06.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:06.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:06.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18773 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0055340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:06.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18773 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0055340).stop 2021-03-09T10:42:06.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:06.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:06.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:06.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18773 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:06.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:42:06.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:06.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:06.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:06.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:06.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:06.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:06.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:06.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.570+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0064d10 con 0x7f63d8003400 2021-03-09T10:42:06.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:06.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8009130 2021-03-09T10:42:06.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:06.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:42:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:06.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:06.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:06.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:06.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:42:06.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:06.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:06.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:06.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:06.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.571+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:06.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:06.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:06.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:42:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:42:06.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:06.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:06.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00147c0 con 0x7f63d8003840 2021-03-09T10:42:06.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:06.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:06.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027fb0 con 0x7f63d8003840 2021-03-09T10:42:06.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:42:06.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:06.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:42:06.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 secure :-1 s=READY pgs=18781 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f00663f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:06.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.572+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:06.585 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:06.578+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52300 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00396f0 con 0x7f63d8003840 2021-03-09T10:42:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.579+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.579+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:06.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.579+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:06.587 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:06.587 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:06.587 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:06.587 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:06.587 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:06.588 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:06.588 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:06.588 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:06.588 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:06.588 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:42:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.579+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.579+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:42:06.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.581+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:42:06.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.710+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:06.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:06.710+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:07.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:07.710+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:07.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:07.710+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:08.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:08.710+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:08.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:08.710+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:09.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:09.710+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:09.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:09.710+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:10.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:10.662+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:10.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:10.711+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:10.711+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:10.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:10.711+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:11.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00153c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:11.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00153c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:11.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 secure :-1 s=READY pgs=18781 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f00663f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:11.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 secure :-1 s=READY pgs=18781 cs=0 l=1 rev1=1 rx=0x7f63f0015e90 tx=0x7f63f00663f0).stop 2021-03-09T10:42:11.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:11.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:11.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00153c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:11.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 unknown :-1 s=CLOSED pgs=18781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:11.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:42:11.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:11.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:11.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:11.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:11.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:11.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:11.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:11.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e990 con 0x7f63d8003400 2021-03-09T10:42:11.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:11.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a080 con 0x7f63d8009130 2021-03-09T10:42:11.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:11.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030070 con 0x7f63d8005e00 2021-03-09T10:42:11.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:11.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:11.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:11.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:11.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.579+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:11.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.580+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:11.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.580+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:42:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.580+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.580+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.580+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.580+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:11.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.580+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:11.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.580+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:11.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:11.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:11.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:11.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:11.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:42:11.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:42:11.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:11.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:11.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007aa40 con 0x7f63d8005e00 2021-03-09T10:42:11.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:11.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:11.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00147c0 con 0x7f63d8005e00 2021-03-09T10:42:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:42:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18796 cs=0 l=1 rev1=1 rx=0x7f63f007c4f0 tx=0x7f63f0029f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:42:11.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.581+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:11.594 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:11.586+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52341 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:11.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.586+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:42:11.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.586+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:11.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.586+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.586+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:11.596 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:11.597 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:11.597 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:11.597 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:11.597 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:11.598 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:42:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.587+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:11.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.587+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:42:11.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:42:11.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.711+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:11.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:11.711+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:12.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:12.711+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:12.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:12.711+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:13.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:13.711+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:13.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:13.711+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:14.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:14.711+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:14.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:14.711+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:15.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:15.663+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:15.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:15.711+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:15.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:15.711+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:16.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:16.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:16.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18796 cs=0 l=1 rev1=1 rx=0x7f63f007c4f0 tx=0x7f63f0029f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:16.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18796 cs=0 l=1 rev1=1 rx=0x7f63f007c4f0 tx=0x7f63f0029f10).stop 2021-03-09T10:42:16.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:16.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:16.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:16.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18796 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:16.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:42:16.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:16.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:16.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:16.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:16.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008f70 con 0x7f63d8003400 2021-03-09T10:42:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3c0 con 0x7f63d8009130 2021-03-09T10:42:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:42:16.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:16.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.587+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:16.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:42:16.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:16.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:16.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:16.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:16.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:42:16.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:42:16.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:16.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:16.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ec20 con 0x7f63d8003840 2021-03-09T10:42:16.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:16.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:16.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006a440 con 0x7f63d8003840 2021-03-09T10:42:16.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:42:16.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.588+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:16.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.590+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 secure :-1 s=READY pgs=18812 cs=0 l=1 rev1=1 rx=0x7f63fc023d80 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:16.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.590+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:42:16.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.590+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:16.602 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:16.595+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52379 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:16.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.597+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:42:16.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.597+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:16.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.597+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:16.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.597+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:16.604 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:16.604 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:16.604 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:16.605 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:16.605 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:16.605 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:16.605 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:16.605 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:16.606 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:16.606 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:16.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:42:16.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.599+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:16.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.599+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:42:16.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.602+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:42:16.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.711+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:16.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:16.711+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:17.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:17.712+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:17.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:17.712+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:18.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:18.711+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:18.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:18.711+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:19.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:19.712+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:19.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:19.712+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:20.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:20.664+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:20.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:20.712+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:20.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:20.712+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:20.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:20.712+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:21.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 secure :-1 s=READY pgs=18812 cs=0 l=1 rev1=1 rx=0x7f63fc023d80 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 secure :-1 s=READY pgs=18812 cs=0 l=1 rev1=1 rx=0x7f63fc023d80 tx=0x7f63fc003a50).stop 2021-03-09T10:42:21.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:21.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:21.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002f410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:21.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 unknown :-1 s=CLOSED pgs=18812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:21.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:42:21.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:21.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:21.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:21.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:21.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:21.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:21.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:21.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T10:42:21.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:21.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029fd0 con 0x7f63d8009130 2021-03-09T10:42:21.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:21.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020b10 con 0x7f63d8005e00 2021-03-09T10:42:21.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:21.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:21.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:21.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:21.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.596+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:21.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.597+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:21.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.597+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:42:21.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.597+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:21.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.597+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:21.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:21.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:21.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:21.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:21.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:21.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:21.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:21.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:21.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:42:21.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:42:21.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:21.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:21.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c990 con 0x7f63d8005e00 2021-03-09T10:42:21.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:21.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:21.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007aa40 con 0x7f63d8005e00 2021-03-09T10:42:21.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:42:21.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.598+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:21.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.600+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:42:21.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.600+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:21.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.600+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18829 cs=0 l=1 rev1=1 rx=0x7f63f007c6c0 tx=0x7f63f0003500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:21.612 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:21.603+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52420 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:21.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0031ec0 con 0x7f63d8005e00 2021-03-09T10:42:21.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.604+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:21.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.604+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:21.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.604+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:21.614 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:21.614 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:21.614 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:21.614 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:21.615 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:21.615 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:21.615 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:21.615 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:21.615 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:21.616 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:21.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.605+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007c350 con 0x7f63d8005e00 2021-03-09T10:42:21.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.605+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:21.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.605+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:42:21.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.606+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005fdb0 con 0x7f63d8005e00 2021-03-09T10:42:21.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.712+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:21.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:21.712+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:22.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:22.712+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:22.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:22.712+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:23.712+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:23.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:23.712+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:24.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:24.712+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:24.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:24.712+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:25.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:25.665+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:25.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:25.713+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:25.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:25.713+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:26.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:26.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:26.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18829 cs=0 l=1 rev1=1 rx=0x7f63f007c6c0 tx=0x7f63f0003500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:26.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18829 cs=0 l=1 rev1=1 rx=0x7f63f007c6c0 tx=0x7f63f0003500).stop 2021-03-09T10:42:26.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:26.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:26.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:26.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:26.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:42:26.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:26.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:26.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:26.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:26.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:26.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:26.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:26.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c410 con 0x7f63d8003400 2021-03-09T10:42:26.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:26.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:42:26.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:26.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3c0 con 0x7f63d8003840 2021-03-09T10:42:26.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:26.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:26.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.604+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:26.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.605+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:26.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:26.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.605+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:26.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.605+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:42:26.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.605+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:26.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.605+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:26.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.605+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:26.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.605+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:26.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.605+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:26.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:26.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:26.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:26.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:26.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:26.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:42:26.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:42:26.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:26.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:26.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006af10 con 0x7f63d8003840 2021-03-09T10:42:26.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:26.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:26.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c990 con 0x7f63d8003840 2021-03-09T10:42:26.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:42:26.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.606+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:26.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.607+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 secure :-1 s=READY pgs=18842 cs=0 l=1 rev1=1 rx=0x7f63f004c810 tx=0x7f63f004c2d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:26.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.607+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:42:26.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.607+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:26.619 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:26.612+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52456 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:26.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.613+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:42:26.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.613+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:26.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.613+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:26.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.613+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:26.621 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:26.622 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:26.622 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:26.622 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:26.622 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:26.623 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:26.624 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.615+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:42:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.615+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.615+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:42:26.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8003840 2021-03-09T10:42:26.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.713+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:26.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:26.713+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:27.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:27.713+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:27.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:27.713+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:28.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:28.713+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:28.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:28.713+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:29.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:29.713+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:29.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:29.713+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:30.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:30.666+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:30.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:30.713+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:30.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:30.713+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:30.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:30.713+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:31.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00153c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:31.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00153c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:31.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 secure :-1 s=READY pgs=18842 cs=0 l=1 rev1=1 rx=0x7f63f004c810 tx=0x7f63f004c2d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:31.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 secure :-1 s=READY pgs=18842 cs=0 l=1 rev1=1 rx=0x7f63f004c810 tx=0x7f63f004c2d0).stop 2021-03-09T10:42:31.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:31.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:31.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00153c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:31.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00153c0 unknown :-1 s=CLOSED pgs=18842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:31.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:42:31.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:31.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:31.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:31.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:31.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:31.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:31.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:31.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T10:42:31.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:31.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029fd0 con 0x7f63d8009130 2021-03-09T10:42:31.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:31.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021eb0 con 0x7f63d8005e00 2021-03-09T10:42:31.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:31.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:31.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.613+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:31.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.614+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:31.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.614+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:31.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.615+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:31.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.615+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:42:31.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.615+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:31.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.615+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:31.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.615+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:31.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.615+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:31.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.615+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:31.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:31.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:31.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:31.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:31.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:31.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:42:31.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:42:31.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:31.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:31.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002cb20 con 0x7f63d8005e00 2021-03-09T10:42:31.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:31.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:31.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006af10 con 0x7f63d8005e00 2021-03-09T10:42:31.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:42:31.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:31.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18856 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f00295a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:31.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:42:31.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.616+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:31.628 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:31.620+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52499 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:31.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.621+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007a460 con 0x7f63d8005e00 2021-03-09T10:42:31.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.621+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:31.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.621+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:31.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.621+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:31.630 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:31.631 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:31.631 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:31.631 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:31.631 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:31.631 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.623+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8005e00 2021-03-09T10:42:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.623+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.623+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:42:31.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.625+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007e250 con 0x7f63d8005e00 2021-03-09T10:42:31.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.713+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:31.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:31.713+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:32.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:32.714+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:32.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:32.714+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:33.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:33.714+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:33.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:33.714+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:34.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:34.714+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:34.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:34.714+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:35.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:35.667+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:35.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:35.714+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:35.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:35.714+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:36.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:36.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:36.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18856 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f00295a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:36.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18856 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f00295a0).stop 2021-03-09T10:42:36.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:36.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:36.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:36.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:36.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:42:36.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:36.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:36.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:36.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:36.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:36.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:36.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:36.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:42:36.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:36.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001d3c0 con 0x7f63d8009130 2021-03-09T10:42:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c410 con 0x7f63d8003840 2021-03-09T10:42:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.620+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:36.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:42:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:36.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:36.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.621+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:36.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:42:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:42:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:36.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:36.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012bf0 con 0x7f63d8003840 2021-03-09T10:42:36.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:36.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:36.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ec20 con 0x7f63d8003840 2021-03-09T10:42:36.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:42:36.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:36.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:42:36.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:36.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.622+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 secure :-1 s=READY pgs=18864 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc071070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:36.635 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:36.628+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52537 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013f10 con 0x7f63d8003840 2021-03-09T10:42:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.628+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.628+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:36.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.628+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:36.637 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:36.638 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:36.638 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:36.638 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:36.638 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:36.638 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:36.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:42:36.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.629+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:36.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.629+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:42:36.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.632+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8003840 2021-03-09T10:42:36.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.714+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:36.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:36.714+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:37.714+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:37.714+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:38.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:38.714+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:38.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:38.714+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:39.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:39.714+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:39.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:39.714+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:40.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:40.668+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:40.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:40.715+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:40.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:40.715+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:40.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:40.715+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:41.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:41.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 secure :-1 s=READY pgs=18864 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc071070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 secure :-1 s=READY pgs=18864 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc071070).stop 2021-03-09T10:42:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:41.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 unknown :-1 s=CLOSED pgs=18864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:42:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:41.634 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:41.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:41.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:41.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:41.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:42:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:41.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8009130 2021-03-09T10:42:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63d8005e00 2021-03-09T10:42:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:41.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:41.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:41.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:41.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:41.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:42:41.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:41.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:41.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:41.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:41.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.629+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:41.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:41.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:41.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:41.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:41.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:41.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:42:41.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:42:41.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:41.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:41.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00656b0 con 0x7f63d8005e00 2021-03-09T10:42:41.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:41.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012bf0 con 0x7f63d8005e00 2021-03-09T10:42:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:42:41.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.630+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.631+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18879 cs=0 l=1 rev1=1 rx=0x7f63fc003a20 tx=0x7f63fc051e90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.631+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:42:41.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.631+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:41.645 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:41.636+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52578 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:41.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.638+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc009520 con 0x7f63d8005e00 2021-03-09T10:42:41.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.638+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:41.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.638+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:41.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.638+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:41.647 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:41.648 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:41.648 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:41.648 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:41.648 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:41.648 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:41.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.640+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8005e00 2021-03-09T10:42:41.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.640+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:41.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.640+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:42:41.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.642+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc009520 con 0x7f63d8005e00 2021-03-09T10:42:41.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.714+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:41.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:41.714+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:42.715+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:42.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:42.715+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:43.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:43.715+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:43.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:43.715+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:44.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:44.715+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:44.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:44.715+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:45.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:45.669+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:45.715+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:45.715+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:46.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:46.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:46.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18879 cs=0 l=1 rev1=1 rx=0x7f63fc003a20 tx=0x7f63fc051e90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:46.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18879 cs=0 l=1 rev1=1 rx=0x7f63fc003a20 tx=0x7f63fc051e90).stop 2021-03-09T10:42:46.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:46.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:46.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:46.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18879 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:46.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:42:46.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:46.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:46.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:46.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:46.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:46.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:46.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:46.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:42:46.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:46.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023cb0 con 0x7f63d8009130 2021-03-09T10:42:46.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:46.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002baf0 con 0x7f63d8003840 2021-03-09T10:42:46.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:46.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:46.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:46.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:46.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:46.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:46.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.637+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:42:46.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:46.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:46.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:46.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:46.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:46.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:42:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:42:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:46.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:46.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017930 con 0x7f63d8003840 2021-03-09T10:42:46.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:46.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:46.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002cb20 con 0x7f63d8003840 2021-03-09T10:42:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:42:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.638+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.639+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:42:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.639+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:46.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.639+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 secure :-1 s=READY pgs=18891 cs=0 l=1 rev1=1 rx=0x7f63f003b2e0 tx=0x7f63f0055290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:46.652 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:46.645+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52616 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.646+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:42:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.646+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.646+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:46.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.646+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:46.654 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:46.655 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:46.656 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.649+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:42:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.649+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:46.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.649+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:42:46.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:42:46.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.715+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:46.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:46.715+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:47.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:47.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:47.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:47.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:48.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:48.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:48.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:48.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:49.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:49.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:49.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:49.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:50.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:50.670+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:50.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:50.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:50.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:50.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:50.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:50.716+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:51.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:42:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 secure :-1 s=READY pgs=18891 cs=0 l=1 rev1=1 rx=0x7f63f003b2e0 tx=0x7f63f0055290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 secure :-1 s=READY pgs=18891 cs=0 l=1 rev1=1 rx=0x7f63f003b2e0 tx=0x7f63f0055290).stop 2021-03-09T10:42:51.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:51.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:51.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068630 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:51.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068630 unknown :-1 s=CLOSED pgs=18891 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:51.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:42:51.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:51.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:51.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:51.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:51.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:51.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:51.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:51.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T10:42:51.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:51.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00281b0 con 0x7f63d8009130 2021-03-09T10:42:51.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:51.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:42:51.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:51.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:51.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:51.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.645+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:51.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.646+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:51.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.646+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:51.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.646+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:42:51.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.646+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:51.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.646+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:51.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.647+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:51.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.647+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:51.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.647+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:51.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:51.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:51.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:51.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:51.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:51.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:42:51.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:42:51.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:51.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:51.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ec20 con 0x7f63d8005e00 2021-03-09T10:42:51.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:51.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:51.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017930 con 0x7f63d8005e00 2021-03-09T10:42:51.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:42:51.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.648+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:51.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.649+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:42:51.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.649+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:51.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.649+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18912 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f004d180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:51.660 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:51.654+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52657 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:51.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.655+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005f8e0 con 0x7f63d8005e00 2021-03-09T10:42:51.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.655+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:51.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.655+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:51.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.655+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:51.662 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:51.663 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:51.663 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:51.663 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:51.663 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:51.663 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:51.664 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:51.664 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:51.664 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:51.664 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:51.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.658+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005f8e0 con 0x7f63d8005e00 2021-03-09T10:42:51.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.658+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:51.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.658+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:42:51.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.659+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:42:51.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:51.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:51.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:52.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:52.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:52.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:52.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:53.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:53.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:53.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:53.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:54.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:54.717+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:54.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:54.717+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:55.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:55.671+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:42:55.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:55.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:55.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:55.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:56.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.654+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:42:56.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.654+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:42:56.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.654+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18912 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f004d180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:42:56.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18912 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f004d180).stop 2021-03-09T10:42:56.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:56.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:42:56.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:42:56.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:42:56.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:42:56.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:56.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:42:56.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:56.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:42:56.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:42:56.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:42:56.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:56.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8003400 2021-03-09T10:42:56.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:56.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:42:56.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:42:56.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00281b0 con 0x7f63d8003840 2021-03-09T10:42:56.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:42:56.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:42:56.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.655+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:56.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.656+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:42:56.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.656+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:42:56.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.657+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:42:56.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.657+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:42:56.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.657+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:42:56.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.657+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:42:56.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:42:56.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:42:56.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:42:56.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:42:56.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:56.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:56.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:42:56.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:42:56.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:42:56.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:42:56.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:42:56.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:56.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0023a90 con 0x7f63d8003840 2021-03-09T10:42:56.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:42:56.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:42:56.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ec20 con 0x7f63d8003840 2021-03-09T10:42:56.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:42:56.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:56.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068ee0 secure :-1 s=READY pgs=18930 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0011f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:42:56.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.658+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:42:56.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.659+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:42:56.672 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:42:56.661+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52695 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:42:56.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f005d920 con 0x7f63d8003840 2021-03-09T10:42:56.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.662+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:42:56.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.662+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:42:56.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.662+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:42:56.674 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:42:56.674 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:42:56.674 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:56.674 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:42:56.674 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:42:56.675 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:42:56.675 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:42:56.675 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:42:56.675 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:42:56.675 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:42:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.663+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005d920 con 0x7f63d8003840 2021-03-09T10:42:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.663+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:42:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.663+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:42:56.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.665+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036a20 con 0x7f63d8003840 2021-03-09T10:42:56.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:56.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:56.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:57.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:57.717+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:57.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:57.717+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:58.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:58.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:58.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:58.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:42:59.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:59.716+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:42:59.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:42:59.716+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:00.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:00.672+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:00.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:00.717+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:00.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:00.717+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:00.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:00.717+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:01.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:01.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:01.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068ee0 secure :-1 s=READY pgs=18930 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0011f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:01.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068ee0 secure :-1 s=READY pgs=18930 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0011f50).stop 2021-03-09T10:43:01.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:01.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:01.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068ee0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:01.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068ee0 unknown :-1 s=CLOSED pgs=18930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:01.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:43:01.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:01.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:01.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:01.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:01.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:01.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:01.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:01.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T10:43:01.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:01.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00275d0 con 0x7f63d8009130 2021-03-09T10:43:01.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:01.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021610 con 0x7f63d8005e00 2021-03-09T10:43:01.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:01.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:01.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:01.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:01.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.662+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:01.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.663+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:01.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.663+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:43:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.663+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.663+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:01.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.663+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:01.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:43:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:43:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:01.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:01.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f410 con 0x7f63d8005e00 2021-03-09T10:43:01.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:01.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:01.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0023a90 con 0x7f63d8005e00 2021-03-09T10:43:01.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:43:01.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.664+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:01.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.665+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18935 cs=0 l=1 rev1=1 rx=0x7f63f00214b0 tx=0x7f63f0066310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:01.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.665+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:43:01.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.665+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:01.677 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:01.669+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52736 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0021520 con 0x7f63d8005e00 2021-03-09T10:43:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.670+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:01.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.670+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.670+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:01.679 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:01.680 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:01.681 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.671+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00212f0 con 0x7f63d8005e00 2021-03-09T10:43:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.671+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:01.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.671+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:43:01.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.675+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8005e00 2021-03-09T10:43:01.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.717+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:01.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:01.717+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:02.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:02.717+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:02.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:02.717+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:03.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:03.717+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:03.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:03.717+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:04.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:04.718+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:04.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:04.718+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:05.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:05.673+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:05.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:05.718+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:05.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:05.718+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:06.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:06.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:06.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18935 cs=0 l=1 rev1=1 rx=0x7f63f00214b0 tx=0x7f63f0066310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:06.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18935 cs=0 l=1 rev1=1 rx=0x7f63f00214b0 tx=0x7f63f0066310).stop 2021-03-09T10:43:06.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:06.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:06.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:06.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:06.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:43:06.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:06.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:06.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:06.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:06.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:06.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:06.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:06.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:43:06.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:06.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8009130 2021-03-09T10:43:06.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:06.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:43:06.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:06.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:06.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:06.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:06.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.670+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:06.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:06.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:43:06.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:06.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:06.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:06.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:06.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:06.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:06.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:06.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:43:06.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:43:06.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.671+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c1d0 con 0x7f63d8003840 2021-03-09T10:43:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.672+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.672+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:06.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.672+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f410 con 0x7f63d8003840 2021-03-09T10:43:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.672+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:43:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.672+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.672+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:43:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.672+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:06.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.672+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 secure :-1 s=READY pgs=18947 cs=0 l=1 rev1=1 rx=0x7f63f0042ff0 tx=0x7f63f0003570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:06.685 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:06.677+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52774 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:06.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:43:06.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.677+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:06.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.677+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:06.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.677+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:06.687 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:06.687 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:06.688 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:06.689 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:06.689 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:06.689 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:06.689 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:43:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.677+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.677+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:43:06.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.678+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036a20 con 0x7f63d8003840 2021-03-09T10:43:06.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.717+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:06.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:06.717+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:07.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:07.718+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:07.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:07.718+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:08.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:08.718+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:08.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:08.718+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:09.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:09.718+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:09.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:09.718+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:10.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:10.674+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:10.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:10.719+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:10.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:10.719+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:10.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:10.719+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:11.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:11.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:11.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 secure :-1 s=READY pgs=18947 cs=0 l=1 rev1=1 rx=0x7f63f0042ff0 tx=0x7f63f0003570).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:11.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 secure :-1 s=READY pgs=18947 cs=0 l=1 rev1=1 rx=0x7f63f0042ff0 tx=0x7f63f0003570).stop 2021-03-09T10:43:11.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:11.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:11.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ba10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:11.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ba10 unknown :-1 s=CLOSED pgs=18947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:11.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:43:11.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:11.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:11.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:11.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:11.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T10:43:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014ce0 con 0x7f63d8009130 2021-03-09T10:43:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:11.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c9f0 con 0x7f63d8005e00 2021-03-09T10:43:11.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:11.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:43:11.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:11.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:11.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:11.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:11.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.678+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:11.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:43:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:43:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:11.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8005e00 2021-03-09T10:43:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c1d0 con 0x7f63d8005e00 2021-03-09T10:43:11.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:43:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.679+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.680+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:43:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.680+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18965 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:11.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.680+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:11.693 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:11.686+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52815 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:11.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.688+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63d8005e00 2021-03-09T10:43:11.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.688+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:11.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.688+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:11.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.688+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:11.695 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:11.695 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:11.695 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:11.695 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:11.695 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:11.696 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:11.696 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:11.696 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:11.696 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:11.697 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:11.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.692+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:43:11.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.692+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:11.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.692+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:43:11.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.692+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0036a20 con 0x7f63d8005e00 2021-03-09T10:43:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.719+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:11.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:11.719+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:12.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:12.719+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:12.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:12.719+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:13.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:13.719+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:13.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:13.719+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:14.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:14.719+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:14.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:14.719+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:15.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:15.675+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:15.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:15.719+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:15.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:15.719+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:16.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:16.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:16.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18965 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:16.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18965 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f004de00).stop 2021-03-09T10:43:16.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18965 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:43:16.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:16.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:16.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:16.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8003400 2021-03-09T10:43:16.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:16.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:43:16.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:16.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003840 2021-03-09T10:43:16.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:16.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:16.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:43:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.687+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:16.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:16.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:16.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:16.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:16.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:16.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:16.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:43:16.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:43:16.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:16.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e940 con 0x7f63d8003840 2021-03-09T10:43:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00656b0 con 0x7f63d8003840 2021-03-09T10:43:16.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:43:16.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.688+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:16.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.689+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:43:16.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.689+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:16.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.689+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e0c0 secure :-1 s=READY pgs=18976 cs=0 l=1 rev1=1 rx=0x7f63e006aaa0 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:16.702 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:16.694+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52853 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:16.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058a60 con 0x7f63d8003840 2021-03-09T10:43:16.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.694+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:16.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.694+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:16.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.694+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:16.704 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:16.704 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:16.704 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:16.705 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:16.705 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:16.705 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:16.705 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:16.705 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:16.706 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:16.706 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:16.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058a60 con 0x7f63d8003840 2021-03-09T10:43:16.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.694+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:16.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.695+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:43:16.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.695+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0710e0 con 0x7f63d8003840 2021-03-09T10:43:16.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.719+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:16.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:16.719+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:17.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:17.719+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:17.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:17.719+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:18.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:18.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:18.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:18.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:19.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:19.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:19.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:19.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:20.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:20.676+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:20.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:20.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:20.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:20.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:20.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:20.720+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:21.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:21.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:21.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e0c0 secure :-1 s=READY pgs=18976 cs=0 l=1 rev1=1 rx=0x7f63e006aaa0 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:21.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e0c0 secure :-1 s=READY pgs=18976 cs=0 l=1 rev1=1 rx=0x7f63e006aaa0 tx=0x7f63fc04a740).stop 2021-03-09T10:43:21.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:21.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:21.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002e0c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:21.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e0c0 unknown :-1 s=CLOSED pgs=18976 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:21.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:43:21.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:21.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:21.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:21.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:21.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:21.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:21.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:21.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020b10 con 0x7f63d8003400 2021-03-09T10:43:21.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:21.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:43:21.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:21.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c9f0 con 0x7f63d8005e00 2021-03-09T10:43:21.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.694+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:21.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.695+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:21.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.695+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:21.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.695+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:21.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.695+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:21.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.695+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:21.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.695+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:43:21.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.695+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:21.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.695+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:21.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.696+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:21.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.696+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:21.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.696+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:21.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:21.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:21.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:21.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:21.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:21.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:43:21.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:43:21.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:21.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:21.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00162e0 con 0x7f63d8005e00 2021-03-09T10:43:21.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:21.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:21.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e940 con 0x7f63d8005e00 2021-03-09T10:43:21.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:43:21.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:21.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18994 cs=0 l=1 rev1=1 rx=0x7f63e0007070 tx=0x7f63fc0476a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:21.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:43:21.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.697+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:21.713 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:21.701+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52894 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:21.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06fc80 con 0x7f63d8005e00 2021-03-09T10:43:21.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.702+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:21.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.702+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:21.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.702+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:21.714 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:21.714 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:21.715 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:21.715 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:21.715 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:21.716 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:21.716 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:21.716 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:21.716 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:21.716 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:21.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.706+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc078c20 con 0x7f63d8005e00 2021-03-09T10:43:21.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.706+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:21.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.706+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:43:21.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.707+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc06fc80 con 0x7f63d8005e00 2021-03-09T10:43:21.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:21.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:21.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:22.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:22.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:22.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:22.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:23.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:23.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:23.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:23.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:24.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:24.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:24.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:24.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:25.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:25.677+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:25.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:25.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:25.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:25.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:26.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:43:26.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18994 cs=0 l=1 rev1=1 rx=0x7f63e0007070 tx=0x7f63fc0476a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 secure :-1 s=READY pgs=18994 cs=0 l=1 rev1=1 rx=0x7f63e0007070 tx=0x7f63fc0476a0).stop 2021-03-09T10:43:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:26.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a90 unknown :-1 s=CLOSED pgs=18994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:43:26.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:26.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:26.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:26.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:43:26.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:26.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:43:26.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:26.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bce0 con 0x7f63d8003840 2021-03-09T10:43:26.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:26.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:26.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:26.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:26.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.702+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:26.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.703+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:26.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.703+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:43:26.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.703+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:26.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.703+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:26.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.703+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:26.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.703+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:26.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.703+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:26.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:26.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:26.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:26.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:26.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:26.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:43:26.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:43:26.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:26.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:26.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0012bf0 con 0x7f63d8003840 2021-03-09T10:43:26.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:26.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:26.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:43:26.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:43:26.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.704+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:26.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.705+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015320 secure :-1 s=READY pgs=19008 cs=0 l=1 rev1=1 rx=0x7f63f0038400 tx=0x7f63f0011f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:26.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.705+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:43:26.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.705+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:26.717 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:26.708+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52932 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:26.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:43:26.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.709+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.709+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.709+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:26.719 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:26.720 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:26.720 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:26.720 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:26.721 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:26.722 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:26.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.713+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:43:26.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.713+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:26.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.713+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:43:26.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.715+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:43:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:26.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:26.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:27.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:27.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:27.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:27.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:28.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:28.721+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:28.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:28.721+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:29.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:29.721+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:29.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:29.721+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:30.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:30.678+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:30.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:30.721+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:30.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:30.721+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:30.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:30.721+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:31.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:31.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:31.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015320 secure :-1 s=READY pgs=19008 cs=0 l=1 rev1=1 rx=0x7f63f0038400 tx=0x7f63f0011f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:31.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015320 secure :-1 s=READY pgs=19008 cs=0 l=1 rev1=1 rx=0x7f63f0038400 tx=0x7f63f0011f40).stop 2021-03-09T10:43:31.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:31.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:31.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:31.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015320 unknown :-1 s=CLOSED pgs=19008 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:31.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:43:31.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:31.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:31.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:31.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:31.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:31.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:31.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:31.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017980 con 0x7f63d8003400 2021-03-09T10:43:31.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:31.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020b10 con 0x7f63d8009130 2021-03-09T10:43:31.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:31.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8005e00 2021-03-09T10:43:31.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:31.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:31.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:31.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:31.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:31.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.709+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:31.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.710+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:43:31.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.710+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:31.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.710+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:31.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.710+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:31.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.710+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:31.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.710+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:31.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:31.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:31.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:31.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:31.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:31.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:43:31.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:43:31.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:31.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:31.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bf70 con 0x7f63d8005e00 2021-03-09T10:43:31.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:31.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:31.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0012bf0 con 0x7f63d8005e00 2021-03-09T10:43:31.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:43:31.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:31.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:43:31.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:31.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.711+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19017 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0038340).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:31.725 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:31.716+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 52973 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:31.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.717+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00884f0 con 0x7f63d8005e00 2021-03-09T10:43:31.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.717+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:31.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.717+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:31.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.717+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:31.727 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:31.727 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:31.727 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:31.727 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:31.728 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:31.728 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:31.728 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:31.728 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:31.728 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:31.728 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:31.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.717+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:43:31.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.717+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:31.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.717+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:43:31.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.718+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:43:31.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.720+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:31.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:31.720+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:32.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:32.721+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:32.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:32.721+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:33.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:33.721+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:33.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:33.721+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:34.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:34.722+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:34.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:34.722+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:35.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:35.679+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:35.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:35.721+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:35.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:35.721+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:36.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:43:36.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:36.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19017 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0038340).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:36.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19017 cs=0 l=1 rev1=1 rx=0x7f63f006f070 tx=0x7f63f0038340).stop 2021-03-09T10:43:36.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:36.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:36.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:36.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=CLOSED pgs=19017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:36.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:43:36.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:36.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:36.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:36.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:36.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:36.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:36.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:36.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c9f0 con 0x7f63d8003400 2021-03-09T10:43:36.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:36.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8009130 2021-03-09T10:43:36.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:36.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:43:36.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:36.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:36.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:36.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:36.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:43:36.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:36.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.717+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:36.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:36.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:36.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:36.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:36.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:36.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:36.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:36.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:36.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:36.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:36.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:43:36.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:43:36.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:36.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:36.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c8f0 con 0x7f63d8003840 2021-03-09T10:43:36.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:36.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:36.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bf70 con 0x7f63d8003840 2021-03-09T10:43:36.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:43:36.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.718+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:36.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.719+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:43:36.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.719+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:36.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.719+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=19030 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f0013b10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:36.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.721+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:36.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.721+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:36.733 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:36.724+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53021 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:36.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.725+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0031dc0 con 0x7f63d8003840 2021-03-09T10:43:36.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.725+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:36.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.725+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.725+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:36.735 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:36.736 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:36.737 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:36.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0031dc0 con 0x7f63d8003840 2021-03-09T10:43:36.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.728+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:36.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.728+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:43:36.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:36.728+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028d50 con 0x7f63d8003840 2021-03-09T10:43:37.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:37.722+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:37.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:37.722+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:38.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:38.722+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:38.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:38.722+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:39.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:39.722+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:39.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:39.722+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:40.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:40.680+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:40.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:40.722+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:40.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:40.722+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:40.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:40.722+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:41.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.722+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:41.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.722+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:41.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:41.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:41.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=19030 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f0013b10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:41.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=19030 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f0013b10).stop 2021-03-09T10:43:41.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:41.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:41.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:41.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=CLOSED pgs=19030 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:41.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:43:41.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:41.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:41.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:41.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:41.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:41.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:41.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:41.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:43:41.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:41.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017980 con 0x7f63d8009130 2021-03-09T10:43:41.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:41.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8005e00 2021-03-09T10:43:41.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:41.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:41.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:41.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:41.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:41.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:41.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:43:41.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:41.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:41.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:41.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:41.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.725+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:41.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:41.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:41.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:41.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:41.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:41.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:43:41.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:43:41.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:41.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:41.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c440 con 0x7f63d8005e00 2021-03-09T10:43:41.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:41.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:41.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00162e0 con 0x7f63d8005e00 2021-03-09T10:43:41.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:43:41.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.726+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:41.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.727+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:43:41.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.727+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19046 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc0039e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:41.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.727+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:41.738 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:41.732+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53062 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:41.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63d8005e00 2021-03-09T10:43:41.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.733+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:41.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.733+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:41.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.733+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:41.740 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:41.740 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:41.741 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:41.741 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:41.741 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:41.741 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:41.741 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:41.742 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:41.742 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:41.742 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:41.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.735+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0489b0 con 0x7f63d8005e00 2021-03-09T10:43:41.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.735+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:41.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.736+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:43:41.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:41.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8005e00 2021-03-09T10:43:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:42.723+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:42.723+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:43.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:43.723+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:43.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:43.723+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:44.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:44.723+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:44.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:44.723+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:45.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:45.681+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:45.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:45.723+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:45.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:45.723+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:46.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.723+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:46.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.723+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:46.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:43:46.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:46.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19046 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc0039e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:46.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19046 cs=0 l=1 rev1=1 rx=0x7f63e00340b0 tx=0x7f63fc0039e0).stop 2021-03-09T10:43:46.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:46.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:46.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:46.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=CLOSED pgs=19046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:46.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:43:46.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:46.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:46.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:46.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:46.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:46.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:46.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:46.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ded0 con 0x7f63d8003400 2021-03-09T10:43:46.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:46.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:43:46.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:46.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8003840 2021-03-09T10:43:46.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:46.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.733+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:46.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.734+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:46.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.734+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:43:46.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.734+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:46.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.734+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:46.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.734+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:46.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.734+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:46.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.734+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:46.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:46.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:46.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:46.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:46.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:46.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:43:46.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:43:46.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:46.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002f170 con 0x7f63d8003840 2021-03-09T10:43:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c8f0 con 0x7f63d8003840 2021-03-09T10:43:46.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:43:46.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:46.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=19064 cs=0 l=1 rev1=1 rx=0x7f63e0068590 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:46.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:43:46.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.735+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:46.747 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:46.740+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53100 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:46.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.740+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:43:46.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.740+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:46.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.740+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:46.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.740+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:46.749 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:46.749 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:46.749 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:46.749 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:46.749 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:46.750 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:46.750 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:46.750 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:46.750 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:46.751 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0039510 con 0x7f63d8003840 2021-03-09T10:43:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.743+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.743+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:43:46.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:46.744+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8003840 2021-03-09T10:43:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:47.723+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:47.723+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:48.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:48.723+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:48.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:48.723+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:49.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:49.724+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:49.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:49.724+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:50.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:50.682+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:50.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:50.724+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:50.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:50.724+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:50.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:50.724+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:51.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.724+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:51.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.724+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=19064 cs=0 l=1 rev1=1 rx=0x7f63e0068590 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:51.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=19064 cs=0 l=1 rev1=1 rx=0x7f63e0068590 tx=0x7f63f004db20).stop 2021-03-09T10:43:51.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:51.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:51.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:51.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=CLOSED pgs=19064 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:43:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:51.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:51.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:51.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068830 con 0x7f63d8003400 2021-03-09T10:43:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011f70 con 0x7f63d8009130 2021-03-09T10:43:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:51.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:43:51.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:51.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:51.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:51.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:51.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:51.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:51.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:43:51.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:51.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:51.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:51.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:51.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.741+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:51.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:51.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:43:51.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:43:51.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:51.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:51.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0022290 con 0x7f63d8005e00 2021-03-09T10:43:51.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:51.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:51.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c440 con 0x7f63d8005e00 2021-03-09T10:43:51.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:43:51.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.742+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:51.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.743+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19072 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc048a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:51.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.743+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:43:51.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.743+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:51.756 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:51.747+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53141 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:51.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023ca0 con 0x7f63d8005e00 2021-03-09T10:43:51.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.747+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:51.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.747+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:51.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.747+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:51.758 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:51.759 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:51.760 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.748+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f63d8005e00 2021-03-09T10:43:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.748+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.748+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:43:51.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:51.749+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc023ca0 con 0x7f63d8005e00 2021-03-09T10:43:52.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:52.724+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:52.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:52.724+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:53.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:53.724+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:53.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:53.724+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:54.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:54.724+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:54.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:54.724+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:55.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:55.683+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:43:55.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:55.724+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:55.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:55.724+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:56.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.724+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:56.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.724+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.748+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:43:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:43:56.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19072 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc048a30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:43:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19072 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc048a30).stop 2021-03-09T10:43:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:43:56.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:43:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=CLOSED pgs=19072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:43:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:43:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0009390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:43:56.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:43:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:43:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:43:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:56.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c9f0 con 0x7f63d8003400 2021-03-09T10:43:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017980 con 0x7f63d8009130 2021-03-09T10:43:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:43:56.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bce0 con 0x7f63d8003840 2021-03-09T10:43:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:43:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:43:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:43:56.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.749+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:43:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.750+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0009390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:43:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.750+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:43:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.750+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:43:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.750+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:43:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.750+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:43:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.750+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:43:56.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.750+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:43:56.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:43:56.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:56.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:56.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:43:56.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:43:56.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:43:56.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:43:56.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:43:56.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:56.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:43:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:43:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:43:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002f170 con 0x7f63d8003840 2021-03-09T10:43:56.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:43:56.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.751+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:56.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.752+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0009390 secure :-1 s=READY pgs=19089 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006c20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:43:56.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.752+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:43:56.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.752+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:43:56.764 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:43:56.756+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53179 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:43:56.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:43:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.756+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:43:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.756+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:43:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.756+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:43:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:43:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:43:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:43:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:43:56.766 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:43:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:43:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:43:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:43:56.767 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:43:56.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0039330 con 0x7f63d8003840 2021-03-09T10:43:56.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.758+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:43:56.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.758+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:43:56.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:56.760+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8003840 2021-03-09T10:43:57.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:57.725+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:57.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:57.725+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:58.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:58.725+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:58.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:58.725+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:43:59.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:59.725+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:43:59.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:43:59.725+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:00.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:00.684+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:00.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:00.725+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:00.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:00.725+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:00.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:00.725+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:01.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.725+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:01.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.725+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:01.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0009390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:01.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0009390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:01.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0009390 secure :-1 s=READY pgs=19089 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006c20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:01.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0009390 secure :-1 s=READY pgs=19089 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0006c20).stop 2021-03-09T10:44:01.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:01.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:01.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0009390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:01.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0009390 unknown :-1 s=CLOSED pgs=19089 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:01.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:44:01.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:01.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:01.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:01.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:01.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:01.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:01.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:01.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T10:44:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065970 con 0x7f63d8009130 2021-03-09T10:44:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8005e00 2021-03-09T10:44:01.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.756+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:01.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.757+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:01.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.757+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:01.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.757+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:01.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.757+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:01.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.758+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:01.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.758+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:44:01.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.758+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:01.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.758+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:01.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.758+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:01.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.758+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:01.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.758+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:01.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:01.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:01.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:01.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:01.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:01.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:44:01.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:44:01.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:01.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:01.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d390 con 0x7f63d8005e00 2021-03-09T10:44:01.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:01.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:01.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8005e00 2021-03-09T10:44:01.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:44:01.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.759+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:01.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.760+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19105 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f005df90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:01.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.760+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:44:01.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.760+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:01.772 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:01.763+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53219 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:01.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8005e00 2021-03-09T10:44:01.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.764+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:01.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.764+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:01.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.764+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:01.774 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:01.775 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:01.775 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:01.775 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:01.775 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:01.776 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:01.776 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:01.776 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:01.776 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:01.776 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:01.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8005e00 2021-03-09T10:44:01.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.764+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:01.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.764+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:44:01.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:01.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007b2b0 con 0x7f63d8005e00 2021-03-09T10:44:02.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:02.725+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:02.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:02.725+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:03.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:03.725+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:03.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:03.725+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:04.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:04.725+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:04.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:04.725+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:05.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:05.685+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:05.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:05.726+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:05.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:05.726+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:06.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.726+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:06.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.726+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:06.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:06.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19105 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f005df90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19105 cs=0 l=1 rev1=1 rx=0x7f63f0006750 tx=0x7f63f005df90).stop 2021-03-09T10:44:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:06.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:06.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=CLOSED pgs=19105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:06.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:44:06.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:06.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:06.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:06.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:06.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:44:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:06.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8009130 2021-03-09T10:44:06.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:06.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c9f0 con 0x7f63d8003840 2021-03-09T10:44:06.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:06.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:06.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:06.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:06.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.764+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:06.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.765+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:06.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.765+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:44:06.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.765+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:06.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.765+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:06.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.765+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:06.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.765+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:06.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.765+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:06.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:06.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:06.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:06.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:06.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:06.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:44:06.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:44:06.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:06.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:06.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001bf70 con 0x7f63d8003840 2021-03-09T10:44:06.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.767+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:06.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.767+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:06.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.767+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d390 con 0x7f63d8003840 2021-03-09T10:44:06.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.767+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:44:06.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.767+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:06.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.769+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:44:06.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.769+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:06.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.769+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=19122 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0011360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:06.778 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:06.771+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53258 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:06.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.772+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00294e0 con 0x7f63d8003840 2021-03-09T10:44:06.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.772+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:06.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.772+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:06.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.772+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:06.779 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:06.779 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:06.780 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:06.780 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:06.780 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:06.780 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:06.780 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:06.781 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:06.781 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:06.781 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:06.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:44:06.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.775+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:06.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.775+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:44:06.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:06.777+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0028d50 con 0x7f63d8003840 2021-03-09T10:44:07.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:07.726+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:07.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:07.726+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:08.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:08.726+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:08.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:08.726+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:09.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:09.726+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:09.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:09.726+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:10.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:10.686+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:10.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:10.726+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:10.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:10.726+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:10.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:10.726+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:11.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.726+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:11.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.726+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:11.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:11.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:11.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=19122 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0011360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:11.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=19122 cs=0 l=1 rev1=1 rx=0x7f63f0009f60 tx=0x7f63f0011360).stop 2021-03-09T10:44:11.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:11.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:11.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:11.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=CLOSED pgs=19122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:11.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:44:11.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:11.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:11.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:11.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:11.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:11.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:11.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:11.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065970 con 0x7f63d8003400 2021-03-09T10:44:11.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:11.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8009130 2021-03-09T10:44:11.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:11.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8005e00 2021-03-09T10:44:11.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.771+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:11.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:11.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.772+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:11.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.772+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:11.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.772+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:11.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.773+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:11.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.773+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:44:11.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.773+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:11.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.773+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:11.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:11.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:11.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:11.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:11.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:11.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:11.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:11.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:11.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:44:11.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:44:11.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:11.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:11.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8005e00 2021-03-09T10:44:11.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:11.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:11.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0022290 con 0x7f63d8005e00 2021-03-09T10:44:11.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:44:11.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.774+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:11.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.775+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:44:11.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.775+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19138 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc023c60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:11.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.775+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:11.785 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:11.778+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53297 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:11.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fe80 con 0x7f63d8005e00 2021-03-09T10:44:11.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.779+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:11.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.779+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:11.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.779+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:11.786 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:11.787 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:11.787 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:11.787 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:11.787 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:11.787 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:11.788 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.780+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fe80 con 0x7f63d8005e00 2021-03-09T10:44:11.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.780+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:11.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.780+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:44:11.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:11.781+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc032180 con 0x7f63d8005e00 2021-03-09T10:44:12.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:12.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:12.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:12.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:13.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:13.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:13.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:13.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:14.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:14.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:14.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:14.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:15.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:15.687+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:15.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:15.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:15.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:15.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:16.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:16.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:16.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:16.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:16.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19138 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc023c60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:16.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19138 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc023c60).stop 2021-03-09T10:44:16.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:16.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:16.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:16.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=CLOSED pgs=19138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:16.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:44:16.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:16.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:16.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:16.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:16.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:16.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:16.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:16.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c490 con 0x7f63d8003400 2021-03-09T10:44:16.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:16.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015c00 con 0x7f63d8009130 2021-03-09T10:44:16.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:16.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e900 con 0x7f63d8003840 2021-03-09T10:44:16.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:16.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:16.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:16.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:16.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:16.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:16.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:44:16.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:16.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.779+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:16.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:16.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:16.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:16.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:16.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:16.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:16.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:16.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:16.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:44:16.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:44:16.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:16.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:16.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ab60 con 0x7f63d8003840 2021-03-09T10:44:16.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:16.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:16.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.780+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:44:16.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.781+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:44:16.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.781+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:16.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.781+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:44:16.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.781+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:16.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.781+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 secure :-1 s=READY pgs=19145 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc0324e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:16.792 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:16.785+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53337 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:16.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.787+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04b040 con 0x7f63d8003840 2021-03-09T10:44:16.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.787+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:16.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.787+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:16.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.787+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:16.794 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:16.794 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:16.794 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:16.795 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:16.795 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:16.795 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:16.795 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:16.795 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:16.796 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.790+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04b040 con 0x7f63d8003840 2021-03-09T10:44:16.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.790+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:16.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.790+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:44:16.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:16.791+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc009520 con 0x7f63d8003840 2021-03-09T10:44:17.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:17.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:17.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:17.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:18.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:18.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:18.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:18.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:19.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:19.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:19.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:19.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:20.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:20.687+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:20.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:20.726+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:20.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:20.726+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:20.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:20.726+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:21.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:21.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:21.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 secure :-1 s=READY pgs=19145 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc0324e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 secure :-1 s=READY pgs=19145 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc0324e0).stop 2021-03-09T10:44:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:21.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:21.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027c30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:21.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 unknown :-1 s=CLOSED pgs=19145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:21.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:44:21.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:21.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:21.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:21.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:21.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c9f0 con 0x7f63d8003400 2021-03-09T10:44:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:44:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8005e00 2021-03-09T10:44:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:21.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.785+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:21.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:21.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:44:21.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:21.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:21.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:21.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:21.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:21.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:21.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:21.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:21.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.786+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:21.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:21.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:44:21.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:44:21.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:21.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:21.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a750 con 0x7f63d8005e00 2021-03-09T10:44:21.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:21.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:21.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001bf70 con 0x7f63d8005e00 2021-03-09T10:44:21.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:44:21.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:21.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:44:21.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19156 cs=0 l=1 rev1=1 rx=0x7f63f0006d40 tx=0x7f63f00554e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:21.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.787+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:21.799 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:21.792+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53374 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:21.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.796+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8005e00 2021-03-09T10:44:21.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.796+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:21.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.796+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:21.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.796+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:21.801 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:21.801 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:21.801 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:21.802 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:21.803 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:21.803 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:21.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.797+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0029740 con 0x7f63d8005e00 2021-03-09T10:44:21.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.797+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:21.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.797+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:44:21.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:21.798+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005d2e0 con 0x7f63d8005e00 2021-03-09T10:44:22.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:22.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:22.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:22.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:23.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:23.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:23.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:23.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:24.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:24.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:24.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:24.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:25.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:25.688+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:25.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:25.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:25.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:25.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:26.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.727+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:26.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.727+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:26.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:26.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:26.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19156 cs=0 l=1 rev1=1 rx=0x7f63f0006d40 tx=0x7f63f00554e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:26.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 secure :-1 s=READY pgs=19156 cs=0 l=1 rev1=1 rx=0x7f63f0006d40 tx=0x7f63f00554e0).stop 2021-03-09T10:44:26.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:26.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:26.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:26.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079a20 unknown :-1 s=CLOSED pgs=19156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:26.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:44:26.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:26.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:26.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:26.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:26.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:26.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:26.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:26.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T10:44:26.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:26.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:44:26.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:26.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e560 con 0x7f63d8003840 2021-03-09T10:44:26.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.792+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:26.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:26.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:26.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:26.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:26.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:26.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:44:26.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:26.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:26.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.793+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:26.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:44:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:44:26.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:26.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:26.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0020ef0 con 0x7f63d8003840 2021-03-09T10:44:26.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:26.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a750 con 0x7f63d8003840 2021-03-09T10:44:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:44:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.794+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.795+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 secure :-1 s=READY pgs=19171 cs=0 l=1 rev1=1 rx=0x7f63f0029000 tx=0x7f63f0006680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.795+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:44:26.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.795+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:26.808 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:26.800+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53416 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:26.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.802+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63d8003840 2021-03-09T10:44:26.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.802+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:26.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.802+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:26.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.802+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:26.810 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:26.810 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:26.810 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:26.810 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:26.811 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:26.811 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:26.811 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:26.811 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:26.812 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:26.812 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:26.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.804+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:44:26.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.804+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:26.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.804+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:44:26.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:26.806+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:44:27.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:27.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:27.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:27.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:28.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:28.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:28.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:28.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:29.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:29.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:29.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:30.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:30.689+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:30.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:30.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:30.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:30.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:30.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:30.728+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:31.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:31.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:31.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:31.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:31.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 secure :-1 s=READY pgs=19171 cs=0 l=1 rev1=1 rx=0x7f63f0029000 tx=0x7f63f0006680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:31.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 secure :-1 s=READY pgs=19171 cs=0 l=1 rev1=1 rx=0x7f63f0029000 tx=0x7f63f0006680).stop 2021-03-09T10:44:31.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:31.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:31.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027c30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027c30 unknown :-1 s=CLOSED pgs=19171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:44:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:31.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:31.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:31.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:31.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:31.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:44:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023c70 con 0x7f63d8009130 2021-03-09T10:44:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:31.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065970 con 0x7f63d8005e00 2021-03-09T10:44:31.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:31.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:31.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:31.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:31.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:31.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.801+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:31.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.802+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:44:31.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.802+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:31.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.802+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:31.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:31.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:31.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:31.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:31.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:31.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:44:31.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:44:31.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:31.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e8b0 con 0x7f63d8005e00 2021-03-09T10:44:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0020ef0 con 0x7f63d8005e00 2021-03-09T10:44:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:44:31.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.803+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:31.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.804+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:44:31.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.804+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:31.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.804+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19186 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:31.818 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:31.808+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53453 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:31.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.809+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:44:31.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.809+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:31.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.809+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:31.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.809+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:31.819 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:31.820 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:31.820 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:31.820 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:31.820 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:31.821 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:31.821 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:31.821 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:31.821 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:31.821 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:31.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.812+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:44:31.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.812+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:31.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.812+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:44:31.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:31.813+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8005e00 2021-03-09T10:44:32.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:32.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:32.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:32.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:33.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:33.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:33.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:33.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:34.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:34.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:34.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:34.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:35.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:35.690+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:35.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:35.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:35.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:35.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:36.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.728+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:36.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.728+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:36.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19186 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19186 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f0013b80).stop 2021-03-09T10:44:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:36.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:44:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:36.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:36.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:36.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:44:36.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:36.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068150 con 0x7f63d8009130 2021-03-09T10:44:36.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:36.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e560 con 0x7f63d8003840 2021-03-09T10:44:36.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:36.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.808+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.809+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.809+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.809+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:44:36.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.809+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:36.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.809+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:36.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.810+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:36.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.810+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.810+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:36.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:36.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:36.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:36.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:44:36.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:44:36.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:36.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:36.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024c30 con 0x7f63d8003840 2021-03-09T10:44:36.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:36.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:36.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e8b0 con 0x7f63d8003840 2021-03-09T10:44:36.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:44:36.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.811+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:36.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.812+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ae0 secure :-1 s=READY pgs=19204 cs=0 l=1 rev1=1 rx=0x7f63f0033040 tx=0x7f63f00884e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:36.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.812+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:44:36.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.812+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:36.824 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:36.816+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53495 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:36.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.816+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:44:36.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.816+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:36.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.816+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:36.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.816+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:36.826 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:36.827 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:36.828 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:36.828 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:36.828 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:36.828 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:36.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:44:36.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.817+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:36.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.817+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:44:36.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:36.819+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007a590 con 0x7f63d8003840 2021-03-09T10:44:37.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:37.729+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:37.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:37.729+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:38.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:38.729+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:38.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:38.729+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:39.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:39.729+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:39.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:39.729+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:40.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:40.691+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:40.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:40.729+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:40.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:40.729+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:40.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:40.729+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:41.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.729+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:41.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.729+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:41.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:44:41.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:41.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ae0 secure :-1 s=READY pgs=19204 cs=0 l=1 rev1=1 rx=0x7f63f0033040 tx=0x7f63f00884e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ae0 secure :-1 s=READY pgs=19204 cs=0 l=1 rev1=1 rx=0x7f63f0033040 tx=0x7f63f00884e0).stop 2021-03-09T10:44:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:41.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065ae0 unknown :-1 s=CLOSED pgs=19204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:41.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:44:41.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:41.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:41.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:41.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:41.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:41.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:41.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:41.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T10:44:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8009130 2021-03-09T10:44:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:41.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:44:41.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:41.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:44:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.817+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:41.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:41.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:41.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:41.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:41.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:41.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:41.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:44:41.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:44:41.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:41.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:41.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a750 con 0x7f63d8005e00 2021-03-09T10:44:41.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:41.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:41.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ab60 con 0x7f63d8005e00 2021-03-09T10:44:41.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.818+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:44:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.819+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.819+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19213 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.820+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:44:41.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.820+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:41.832 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:41.824+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53534 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.824+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc066ee0 con 0x7f63d8005e00 2021-03-09T10:44:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.824+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.824+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.824+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:41.834 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:41.835 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:41.836 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:41.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.827+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc066ee0 con 0x7f63d8005e00 2021-03-09T10:44:41.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.827+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:41.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.827+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:44:41.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:41.828+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8005e00 2021-03-09T10:44:42.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:42.730+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:42.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:42.730+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:43.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:43.729+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:43.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:43.729+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:44.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:44.730+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:44.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:44.730+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:45.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:45.692+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:45.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:45.730+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:45.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:45.730+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:46.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.730+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:46.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.730+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.824+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:46.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.824+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.824+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19213 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.824+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19213 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63fc009990).stop 2021-03-09T10:44:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:46.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:44:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:46.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:46.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:46.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:46.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:46.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:46.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:46.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e560 con 0x7f63d8003400 2021-03-09T10:44:46.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:46.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8009130 2021-03-09T10:44:46.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:46.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065970 con 0x7f63d8003840 2021-03-09T10:44:46.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:46.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:46.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:46.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:46.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.825+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:46.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.826+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:46.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.826+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:44:46.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.826+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:46.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.826+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:46.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.826+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:46.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.826+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:46.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.826+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:46.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:46.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:46.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:46.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:46.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:46.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:44:46.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:44:46.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:46.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:46.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0009750 con 0x7f63d8003840 2021-03-09T10:44:46.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:46.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:46.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024c30 con 0x7f63d8003840 2021-03-09T10:44:46.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:44:46.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.827+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:46.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.828+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 secure :-1 s=READY pgs=19229 cs=0 l=1 rev1=1 rx=0x7f63f0012000 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:46.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.828+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:44:46.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.828+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:46.840 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:46.832+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53574 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:46.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.833+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038e80 con 0x7f63d8003840 2021-03-09T10:44:46.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.833+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:46.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.833+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:46.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.833+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:46.842 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:46.842 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:46.842 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:46.842 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:46.842 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:46.843 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:46.843 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:46.843 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:46.843 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:46.843 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:46.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.837+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0038e80 con 0x7f63d8003840 2021-03-09T10:44:46.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.837+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:46.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.837+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:44:46.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:46.838+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8003840 2021-03-09T10:44:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:47.730+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:47.730+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:48.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:48.730+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:48.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:48.730+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:49.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:49.731+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:49.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:49.731+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:50.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:50.693+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:50.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:50.730+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:50.730+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:50.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:50.730+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:51.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.731+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:51.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.731+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:51.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:51.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:51.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 secure :-1 s=READY pgs=19229 cs=0 l=1 rev1=1 rx=0x7f63f0012000 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:51.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 secure :-1 s=READY pgs=19229 cs=0 l=1 rev1=1 rx=0x7f63f0012000 tx=0x7f63f004db20).stop 2021-03-09T10:44:51.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:51.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:51.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:51.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 unknown :-1 s=CLOSED pgs=19229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:51.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:44:51.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:51.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:51.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:51.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:51.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:51.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:51.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:51.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:44:51.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:51.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:44:51.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:51.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e0d0 con 0x7f63d8005e00 2021-03-09T10:44:51.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.832+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:51.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.833+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:51.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.833+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:51.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.833+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:51.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.833+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:51.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.833+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:51.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.833+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:44:51.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.833+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:51.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.833+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:51.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.834+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:51.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.834+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:51.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.834+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:51.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:51.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:51.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:51.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:51.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:51.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:44:51.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:44:51.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:51.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:51.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d390 con 0x7f63d8005e00 2021-03-09T10:44:51.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:51.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:51.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0009750 con 0x7f63d8005e00 2021-03-09T10:44:51.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:44:51.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.835+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:51.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.836+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:44:51.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.836+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:51.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.836+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19246 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f0038f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:51.847 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:51.840+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53613 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:51.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.840+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:44:51.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.840+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:51.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.840+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:51.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.840+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:51.849 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:51.849 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:51.849 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:51.849 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:51.850 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:51.850 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:51.850 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:51.850 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:51.850 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:51.851 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:51.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.843+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:44:51.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.843+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:51.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.843+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:44:51.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:51.844+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8005e00 2021-03-09T10:44:52.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:52.731+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:52.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:52.731+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:53.731+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:53.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:53.731+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:54.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:54.731+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:54.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:54.731+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:55.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:55.694+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:44:55.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:55.731+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:55.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:55.731+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:56.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.731+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:56.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.731+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:56.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:44:56.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:44:56.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19246 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f0038f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:44:56.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19246 cs=0 l=1 rev1=1 rx=0x7f63f0027ff0 tx=0x7f63f0038f60).stop 2021-03-09T10:44:56.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:56.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:44:56.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:44:56.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:44:56.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:44:56.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:56.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:44:56.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:56.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:44:56.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:44:56.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:44:56.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:56.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:44:56.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:56.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8009130 2021-03-09T10:44:56.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:44:56.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010e00 con 0x7f63d8003840 2021-03-09T10:44:56.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:44:56.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:44:56.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.841+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:56.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:44:56.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:44:56.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:44:56.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.842+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:44:56.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.842+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:44:56.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.842+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:44:56.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:44:56.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:44:56.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:44:56.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:44:56.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:56.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:56.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:44:56.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:44:56.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:44:56.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:44:56.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:44:56.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:56.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00303e0 con 0x7f63d8003840 2021-03-09T10:44:56.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:44:56.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:44:56.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d390 con 0x7f63d8003840 2021-03-09T10:44:56.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:44:56.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.843+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:56.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.844+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:44:56.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.844+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:44:56.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.844+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022290 secure :-1 s=READY pgs=19260 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:44:56.858 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:44:56.849+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53653 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:44:56.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.849+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:44:56.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.849+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:44:56.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.849+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:44:56.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.849+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:44:56.860 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:44:56.860 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:44:56.860 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:56.861 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:44:56.861 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:44:56.861 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:44:56.861 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:44:56.861 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:44:56.862 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:44:56.862 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:44:56.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:44:56.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.850+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:44:56.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.850+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:44:56.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:56.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8003840 2021-03-09T10:44:57.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:57.732+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:57.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:57.732+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:58.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:58.732+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:58.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:58.732+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:44:59.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:59.732+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:44:59.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:44:59.732+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:00.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:00.695+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:00.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:00.732+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:00.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:00.732+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:00.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:00.732+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:01.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.732+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:01.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.732+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:01.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.849+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:01.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.849+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:01.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022290 secure :-1 s=READY pgs=19260 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:01.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.849+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022290 secure :-1 s=READY pgs=19260 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0013b80).stop 2021-03-09T10:45:01.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.849+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:01.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.849+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:01.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.849+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022290 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:01.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.849+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022290 unknown :-1 s=CLOSED pgs=19260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:01.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.849+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:45:01.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:01.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:01.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:01.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:01.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:01.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:01.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:01.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011d70 con 0x7f63d8003400 2021-03-09T10:45:01.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:01.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a580 con 0x7f63d8009130 2021-03-09T10:45:01.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:01.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065970 con 0x7f63d8005e00 2021-03-09T10:45:01.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:01.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:01.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:01.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:01.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:01.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:01.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:45:01.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:01.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.850+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:01.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.851+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:01.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.851+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:01.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.851+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:01.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:01.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:01.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:01.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:01.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:01.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:01.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:01.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:01.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:01.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013d00 con 0x7f63d8005e00 2021-03-09T10:45:01.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:01.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:01.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00303e0 con 0x7f63d8005e00 2021-03-09T10:45:01.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:01.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.852+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:01.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.853+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:45:01.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.853+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:01.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.853+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19270 cs=0 l=1 rev1=1 rx=0x7f63f00555e0 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:01.868 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:01.857+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53682 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:01.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007c350 con 0x7f63d8005e00 2021-03-09T10:45:01.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.858+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:01.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.858+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:01.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.858+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:01.870 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:01.870 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:01.870 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:01.870 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:01.871 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:01.871 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:01.871 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:01.872 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:01.872 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:01.872 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:01.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.861+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007e640 con 0x7f63d8005e00 2021-03-09T10:45:01.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.861+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:01.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.861+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:45:01.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:01.864+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0039be0 con 0x7f63d8005e00 2021-03-09T10:45:02.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:02.732+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:02.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:02.732+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:03.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:03.732+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:03.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:03.732+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:04.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:04.732+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:04.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:04.732+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:05.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:05.696+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:05.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:05.733+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:05.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:05.733+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:06.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.732+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:06.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.732+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:06.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:06.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:06.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19270 cs=0 l=1 rev1=1 rx=0x7f63f00555e0 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:06.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19270 cs=0 l=1 rev1=1 rx=0x7f63f00555e0 tx=0x7f63f004de00).stop 2021-03-09T10:45:06.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:06.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:06.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:06.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:06.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:45:06.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:06.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:06.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:06.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:06.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:06.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:06.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:06.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016fd0 con 0x7f63d8003400 2021-03-09T10:45:06.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:06.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e560 con 0x7f63d8009130 2021-03-09T10:45:06.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:06.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:45:06.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:06.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:06.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:06.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.858+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.859+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:45:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.859+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:06.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.859+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:06.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:06.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:06.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.859+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:06.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:06.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:06.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:06.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:06.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:06.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:06.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:06.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:06.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:06.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0009750 con 0x7f63d8003840 2021-03-09T10:45:06.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:06.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:06.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013d00 con 0x7f63d8003840 2021-03-09T10:45:06.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:06.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.860+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:06.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.861+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba90 secure :-1 s=READY pgs=19282 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007e720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:06.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.861+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:45:06.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.861+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:06.875 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:06.865+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53732 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:06.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:45:06.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.866+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:06.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.866+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:06.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.866+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:06.877 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:06.877 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:06.877 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:06.877 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:06.878 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:06.878 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:06.878 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:06.878 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:06.879 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:06.879 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:06.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.870+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:45:06.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.870+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:06.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.870+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:45:06.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:06.871+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00112f0 con 0x7f63d8003840 2021-03-09T10:45:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:07.733+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:07.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:07.733+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:08.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:08.733+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:08.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:08.733+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:09.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:09.733+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:09.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:09.733+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:10.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:10.697+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:10.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:10.733+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:10.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:10.733+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:10.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:10.733+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:11.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.734+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:11.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.734+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:11.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:45:11.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:11.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba90 secure :-1 s=READY pgs=19282 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007e720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:11.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba90 secure :-1 s=READY pgs=19282 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007e720).stop 2021-03-09T10:45:11.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:11.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:11.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002ba90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002ba90 unknown :-1 s=CLOSED pgs=19282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:45:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:11.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0012ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:11.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011d70 con 0x7f63d8003400 2021-03-09T10:45:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016fd0 con 0x7f63d8009130 2021-03-09T10:45:11.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8005e00 2021-03-09T10:45:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:11.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.866+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:11.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.867+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0012ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.867+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:45:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.867+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.867+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:11.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.867+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.867+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.867+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:11.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:11.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024c30 con 0x7f63d8005e00 2021-03-09T10:45:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:11.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0009750 con 0x7f63d8005e00 2021-03-09T10:45:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.868+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.869+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:45:11.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.869+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0012ff0 secure :-1 s=READY pgs=19301 cs=0 l=1 rev1=1 rx=0x7f63f002f7a0 tx=0x7f63f0006680).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:11.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.869+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:11.881 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:11.873+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53761 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63d8005e00 2021-03-09T10:45:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.873+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.873+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:11.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.873+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:11.883 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:11.883 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:11.883 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:11.884 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:11.885 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.876+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002f3b0 con 0x7f63d8005e00 2021-03-09T10:45:11.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.876+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:11.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.876+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:45:11.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:11.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8005e00 2021-03-09T10:45:12.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:12.734+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:12.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:12.734+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:13.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:13.734+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:13.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:13.734+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:14.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:14.734+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:14.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:14.734+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:15.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:15.698+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:15.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:15.734+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:15.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:15.734+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:16.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.734+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:16.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.734+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0012ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:45:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0012ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:16.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0012ff0 secure :-1 s=READY pgs=19301 cs=0 l=1 rev1=1 rx=0x7f63f002f7a0 tx=0x7f63f0006680).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0012ff0 secure :-1 s=READY pgs=19301 cs=0 l=1 rev1=1 rx=0x7f63f002f7a0 tx=0x7f63f0006680).stop 2021-03-09T10:45:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0012ff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:16.877 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0012ff0 unknown :-1 s=CLOSED pgs=19301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:16.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:45:16.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:16.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:16.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:16.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:16.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:16.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:16.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:16.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b830 con 0x7f63d8003400 2021-03-09T10:45:16.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:16.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:45:16.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:16.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d710 con 0x7f63d8003840 2021-03-09T10:45:16.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:16.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:16.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:16.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:16.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.873+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:16.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.874+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:16.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.874+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:45:16.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.874+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:16.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.874+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:16.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.874+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:16.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.874+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:16.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.874+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:16.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:16.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:16.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:16.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:16.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:16.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:16.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:16.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:16.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:16.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d390 con 0x7f63d8003840 2021-03-09T10:45:16.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:16.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:16.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024c30 con 0x7f63d8003840 2021-03-09T10:45:16.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:16.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.875+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.876+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b020 secure :-1 s=READY pgs=19307 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f00733b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.876+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:45:16.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.876+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:16.889 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:16.880+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53811 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:16.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:45:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.881+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.881+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.881+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:16.890 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:16.891 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:16.891 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:16.891 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:16.891 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:16.891 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:16.892 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:16.892 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:16.892 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:16.892 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:16.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005d2e0 con 0x7f63d8003840 2021-03-09T10:45:16.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.881+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:16.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.881+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:45:16.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:16.882+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63d8003840 2021-03-09T10:45:17.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:17.734+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:17.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:17.734+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:18.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:18.734+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:18.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:18.734+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:19.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:19.735+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:19.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:19.735+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:20.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:20.699+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:20.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:20.734+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:20.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:20.734+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:20.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:20.734+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:21.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.735+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:21.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.735+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:21.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:21.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:21.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b020 secure :-1 s=READY pgs=19307 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f00733b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:21.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b020 secure :-1 s=READY pgs=19307 cs=0 l=1 rev1=1 rx=0x7f63f0036ee0 tx=0x7f63f00733b0).stop 2021-03-09T10:45:21.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:21.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:21.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001b020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:21.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b020 unknown :-1 s=CLOSED pgs=19307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:21.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:45:21.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:21.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:21.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:21.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017c30 con 0x7f63d8003400 2021-03-09T10:45:21.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:21.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002bff0 con 0x7f63d8009130 2021-03-09T10:45:21.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:21.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8005e00 2021-03-09T10:45:21.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:21.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:21.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:21.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:21.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.881+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:21.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.882+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:21.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.882+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:45:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.882+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.882+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.882+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.882+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:21.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.882+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:21.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:21.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:21.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:21.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:21.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:21.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:21.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:21.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:21.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:21.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8005e00 2021-03-09T10:45:21.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:21.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:21.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d390 con 0x7f63d8005e00 2021-03-09T10:45:21.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:21.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.883+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:21.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.884+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19320 cs=0 l=1 rev1=1 rx=0x7f63f0088a70 tx=0x7f63f0081410).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:21.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.884+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:45:21.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.884+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:21.897 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:21.888+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53850 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:21.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.888+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:45:21.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.888+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:21.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.888+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:21.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.888+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:21.899 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:21.899 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:21.900 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:21.900 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:21.900 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:21.900 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:21.900 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:21.901 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:45:21.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.889+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:21.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.889+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:45:21.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:21.890+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8005e00 2021-03-09T10:45:22.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:22.735+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:22.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:22.735+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:23.735+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:23.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:23.735+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:24.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:24.735+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:24.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:24.735+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:25.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:25.700+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:25.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:25.735+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:25.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:25.735+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.736+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:26.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.736+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:26.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:45:26.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:26.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19320 cs=0 l=1 rev1=1 rx=0x7f63f0088a70 tx=0x7f63f0081410).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:26.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19320 cs=0 l=1 rev1=1 rx=0x7f63f0088a70 tx=0x7f63f0081410).stop 2021-03-09T10:45:26.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:26.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:26.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:26.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:26.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:45:26.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:26.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:26.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:26.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:26.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:26.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:26.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:26.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016fd0 con 0x7f63d8003400 2021-03-09T10:45:26.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:26.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8009130 2021-03-09T10:45:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c7b0 con 0x7f63d8003840 2021-03-09T10:45:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:26.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:26.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:26.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:26.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.889+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:26.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.890+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:26.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.890+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:45:26.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.890+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:26.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.890+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:26.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:26.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:26.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.890+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:26.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:26.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:26.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:26.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:26.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:26.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:26.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:26.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00230b0 con 0x7f63d8003840 2021-03-09T10:45:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:26.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:45:26.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:26.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:26.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=19338 cs=0 l=1 rev1=1 rx=0x7f63e0018180 tx=0x7f63f0055550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:26.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:45:26.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.891+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:26.904 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:26.897+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53890 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:26.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.898+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:45:26.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.898+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:26.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.898+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:26.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.898+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:26.906 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:26.906 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:26.906 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:26.906 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:26.906 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:26.907 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:26.907 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:26.907 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:26.907 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:26.908 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:26.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.901+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:45:26.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.901+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:26.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.901+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:45:26.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:26.904+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00288e0 con 0x7f63d8003840 2021-03-09T10:45:27.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:27.736+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:27.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:27.736+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:28.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:28.736+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:28.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:28.736+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:29.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:29.736+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:29.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:29.736+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:30.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:30.701+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:30.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:30.736+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:30.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:30.736+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:30.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:30.736+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:31.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.736+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:31.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.736+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:31.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:31.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:31.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=19338 cs=0 l=1 rev1=1 rx=0x7f63e0018180 tx=0x7f63f0055550).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:31.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 secure :-1 s=READY pgs=19338 cs=0 l=1 rev1=1 rx=0x7f63e0018180 tx=0x7f63f0055550).stop 2021-03-09T10:45:31.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:31.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:31.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00289c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:31.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00289c0 unknown :-1 s=CLOSED pgs=19338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:31.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:45:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:31.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:31.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:31.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:31.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:45:31.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:31.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065970 con 0x7f63d8009130 2021-03-09T10:45:31.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:31.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8005e00 2021-03-09T10:45:31.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.897+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:31.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.898+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:31.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.898+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:31.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.898+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:31.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.898+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:45:31.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.898+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:31.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.898+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:31.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.898+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:31.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.898+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:31.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.899+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:31.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.899+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:31.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.899+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:31.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:31.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:31.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:31.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:31.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:31.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:31.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:31.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:31.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:31.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0020ef0 con 0x7f63d8005e00 2021-03-09T10:45:31.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:31.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:31.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00230b0 con 0x7f63d8005e00 2021-03-09T10:45:31.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:31.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:31.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001bab0 secure :-1 s=READY pgs=19352 cs=0 l=1 rev1=1 rx=0x7f63f002ff30 tx=0x7f63f0042470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:31.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:45:31.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.900+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:31.912 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:31.905+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53929 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:31.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:45:31.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.905+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:31.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.905+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:31.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.905+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:31.914 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:31.914 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:31.914 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:31.914 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:31.915 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:31.915 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:31.915 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:31.915 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:31.915 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:31.916 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:31.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.907+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:45:31.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.907+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:31.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.907+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:45:31.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:31.909+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0088aa0 con 0x7f63d8005e00 2021-03-09T10:45:32.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:32.736+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:32.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:32.736+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:33.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:33.736+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:33.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:33.736+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:34.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:34.737+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:34.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:34.737+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:35.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:35.702+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:35.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:35.737+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:35.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:35.737+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:36.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.736+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:36.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.736+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:36.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:36.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:36.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001bab0 secure :-1 s=READY pgs=19352 cs=0 l=1 rev1=1 rx=0x7f63f002ff30 tx=0x7f63f0042470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:36.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001bab0 secure :-1 s=READY pgs=19352 cs=0 l=1 rev1=1 rx=0x7f63f002ff30 tx=0x7f63f0042470).stop 2021-03-09T10:45:36.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:36.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:36.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e001bab0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:36.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e001bab0 unknown :-1 s=CLOSED pgs=19352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:36.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:45:36.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00290a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:36.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:36.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:36.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:36.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:36.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:36.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:36.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:45:36.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:36.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067490 con 0x7f63d8009130 2021-03-09T10:45:36.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:36.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:45:36.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:36.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:36.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:36.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:36.914 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:36.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.907+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00290a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:36.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.907+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:45:36.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.907+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:36.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.907+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:36.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:36.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:36.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:36.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:36.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:36.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:36.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:36.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:36.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:36.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:36.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:36.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:36.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00303e0 con 0x7f63d8003840 2021-03-09T10:45:36.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:36.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:36.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0020ef0 con 0x7f63d8003840 2021-03-09T10:45:36.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:36.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:36.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:45:36.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.908+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00290a0 secure :-1 s=READY pgs=19366 cs=0 l=1 rev1=1 rx=0x7f63f00385c0 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:36.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.909+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:36.921 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:36.913+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 53969 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:36.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0039740 con 0x7f63d8003840 2021-03-09T10:45:36.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.914+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:36.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.914+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:36.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.914+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:36.923 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:36.923 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:36.923 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:36.923 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:36.923 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:36.924 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:36.924 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:36.924 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:36.924 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:36.924 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:36.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.919+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0039740 con 0x7f63d8003840 2021-03-09T10:45:36.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.919+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:36.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.919+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:45:36.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:36.920+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0088aa0 con 0x7f63d8003840 2021-03-09T10:45:37.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:37.737+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:37.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:37.737+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:38.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:38.737+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:38.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:38.737+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:39.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:39.737+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:39.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:39.737+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:40.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:40.703+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:40.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:40.737+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:40.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:40.737+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:40.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:40.737+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:41.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.738+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:41.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.738+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:41.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00290a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:41.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00290a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:41.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00290a0 secure :-1 s=READY pgs=19366 cs=0 l=1 rev1=1 rx=0x7f63f00385c0 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:41.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00290a0 secure :-1 s=READY pgs=19366 cs=0 l=1 rev1=1 rx=0x7f63f00385c0 tx=0x7f63f000df50).stop 2021-03-09T10:45:41.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:41.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:41.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00290a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:41.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00290a0 unknown :-1 s=CLOSED pgs=19366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:41.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:45:41.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:41.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:41.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:41.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:41.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00017f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:41.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:41.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:41.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020d10 con 0x7f63d8003400 2021-03-09T10:45:41.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:41.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:45:41.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:41.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:45:41.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:41.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:41.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:41.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:41.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:41.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00017f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:41.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:45:41.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:41.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:41.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:41.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:41.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.914+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:41.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:41.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:41.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:41.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:41.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:41.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:41.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:41.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:41.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:41.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b830 con 0x7f63d8005e00 2021-03-09T10:45:41.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:41.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:41.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00303e0 con 0x7f63d8005e00 2021-03-09T10:45:41.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:41.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:41.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:45:41.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:41.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.915+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00017f0 secure :-1 s=READY pgs=19376 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0032360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:41.929 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:41.920+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54008 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:41.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007a430 con 0x7f63d8005e00 2021-03-09T10:45:41.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.922+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:41.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.922+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:41.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.922+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:41.931 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:41.931 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:41.931 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:41.931 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:41.932 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:41.932 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:41.932 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:41.932 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:41.932 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:41.933 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:41.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63d8005e00 2021-03-09T10:45:41.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.922+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:41.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.922+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:45:41.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:41.922+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0023bc0 con 0x7f63d8005e00 2021-03-09T10:45:42.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:42.738+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:42.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:42.738+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:43.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:43.738+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:43.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:43.738+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:44.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:44.738+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:44.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:44.738+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:45.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:45.704+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:45.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:45.738+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:45.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:45.738+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.738+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:46.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.738+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:46.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:46.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00017f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:46.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00017f0 secure :-1 s=READY pgs=19376 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0032360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:46.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00017f0 secure :-1 s=READY pgs=19376 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0032360).stop 2021-03-09T10:45:46.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:46.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:46.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00017f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:46.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00017f0 unknown :-1 s=CLOSED pgs=19376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:46.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:45:46.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:46.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:46.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:46.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:46.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:46.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:46.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:46.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:45:46.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:46.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:45:46.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:46.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c140 con 0x7f63d8003840 2021-03-09T10:45:46.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:46.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:46.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:45:46.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:46.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.921+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:46.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:46.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:46.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:46.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:46.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:46.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:46.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:46.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:46.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:46.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:46.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:46.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:46.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:45:46.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:45:46.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:46.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:46.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017c10 con 0x7f63d8003840 2021-03-09T10:45:46.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:46.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:46.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a750 con 0x7f63d8003840 2021-03-09T10:45:46.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:45:46.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.922+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:46.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.923+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015080 secure :-1 s=READY pgs=19390 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc071150).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:46.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.923+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:45:46.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.923+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:46.935 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:46.928+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54048 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:46.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc023e50 con 0x7f63d8003840 2021-03-09T10:45:46.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.929+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:46.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.929+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:46.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.929+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:46.937 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:46.937 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:46.937 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:46.937 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:46.938 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:46.938 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:46.938 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:46.938 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:46.938 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:46.939 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:46.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023e50 con 0x7f63d8003840 2021-03-09T10:45:46.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.932+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:46.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.932+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:45:46.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:46.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8003840 2021-03-09T10:45:47.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:47.738+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:47.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:47.738+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:48.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:48.739+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:48.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:48.739+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:49.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:49.739+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:49.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:49.739+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:50.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:50.705+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:50.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:50.739+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:50.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:50.739+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:50.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:50.739+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:51.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.738+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:51.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.738+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:51.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:51.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:51.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015080 secure :-1 s=READY pgs=19390 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc071150).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:51.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015080 secure :-1 s=READY pgs=19390 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc071150).stop 2021-03-09T10:45:51.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:51.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:51.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0015080 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:51.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0015080 unknown :-1 s=CLOSED pgs=19390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:51.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:45:51.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:51.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:51.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:51.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:51.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:51.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:51.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:51.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63d8003400 2021-03-09T10:45:51.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:51.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023c70 con 0x7f63d8009130 2021-03-09T10:45:51.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:51.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025110 con 0x7f63d8005e00 2021-03-09T10:45:51.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:51.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:51.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:51.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:51.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:45:51.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:51.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.929+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:51.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:51.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:51.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:45:51.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:45:51.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:51.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:51.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ca10 con 0x7f63d8005e00 2021-03-09T10:45:51.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:51.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:51.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b830 con 0x7f63d8005e00 2021-03-09T10:45:51.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:45:51.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.930+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:51.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.931+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19404 cs=0 l=1 rev1=1 rx=0x7f63f0033040 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:51.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.931+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:45:51.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.931+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:51.945 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:51.936+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54087 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:51.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8005e00 2021-03-09T10:45:51.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.936+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:51.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.936+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:51.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.936+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:51.947 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:51.947 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:51.947 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:51.947 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:51.948 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:51.948 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:51.948 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:51.948 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:51.948 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:51.949 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:51.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.940+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0080e30 con 0x7f63d8005e00 2021-03-09T10:45:51.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.940+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:51.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.940+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:45:51.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:51.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8005e00 2021-03-09T10:45:52.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:52.738+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:52.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:52.739+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:53.739+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:53.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:53.739+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:54.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:54.739+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:54.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:54.739+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:55.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:55.706+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:45:55.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:55.740+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:55.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:55.740+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:56.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.740+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:56.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.740+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:56.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:45:56.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:45:56.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19404 cs=0 l=1 rev1=1 rx=0x7f63f0033040 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:45:56.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19404 cs=0 l=1 rev1=1 rx=0x7f63f0033040 tx=0x7f63f0000bc0).stop 2021-03-09T10:45:56.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:56.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:45:56.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:45:56.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:45:56.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:45:56.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:56.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:45:56.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:56.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:45:56.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:45:56.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:45:56.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:56.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f63d8003400 2021-03-09T10:45:56.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:56.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:45:56.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:45:56.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023c70 con 0x7f63d8003840 2021-03-09T10:45:56.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:45:56.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:45:56.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:56.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:45:56.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.936+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:45:56.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:45:56.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:45:56.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:45:56.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:45:56.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:45:56.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:45:56.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:45:56.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:45:56.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:56.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:56.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:45:56.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:45:56.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:45:56.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:45:56.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:45:56.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:56.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0020ef0 con 0x7f63d8003840 2021-03-09T10:45:56.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:45:56.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:45:56.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017c10 con 0x7f63d8003840 2021-03-09T10:45:56.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:45:56.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.937+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:56.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.938+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022270 secure :-1 s=READY pgs=19418 cs=0 l=1 rev1=1 rx=0x7f63e0066740 tx=0x7f63fc032470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:45:56.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.938+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:45:56.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.938+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:45:56.951 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:45:56.945+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54127 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:45:56.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:45:56.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.945+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:45:56.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.945+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:45:56.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.945+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:45:56.953 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:45:56.954 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:45:56.954 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:56.954 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:45:56.954 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:45:56.954 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:45:56.955 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:45:56.955 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:45:56.955 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:45:56.955 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:45:56.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc023d80 con 0x7f63d8003840 2021-03-09T10:45:56.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.945+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:45:56.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.945+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:45:56.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:56.946+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8003840 2021-03-09T10:45:57.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:57.740+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:57.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:57.740+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:58.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:58.739+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:58.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:58.739+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:45:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:59.740+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:45:59.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:45:59.740+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:00.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:00.707+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:00.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:00.740+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:00.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:00.740+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:00.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:00.740+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:01.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.740+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:01.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.740+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:01.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022270 secure :-1 s=READY pgs=19418 cs=0 l=1 rev1=1 rx=0x7f63e0066740 tx=0x7f63fc032470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022270 secure :-1 s=READY pgs=19418 cs=0 l=1 rev1=1 rx=0x7f63e0066740 tx=0x7f63fc032470).stop 2021-03-09T10:46:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:01.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0022270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0022270 unknown :-1 s=CLOSED pgs=19418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:46:01.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:01.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00104e0 con 0x7f63d8003400 2021-03-09T10:46:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:01.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065970 con 0x7f63d8009130 2021-03-09T10:46:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0020d10 con 0x7f63d8005e00 2021-03-09T10:46:01.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.945+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.946+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:01.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.946+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.946+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.946+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:46:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.946+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:01.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.946+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:01.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:01.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:01.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:01.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:01.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:01.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079dc0 con 0x7f63d8005e00 2021-03-09T10:46:01.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:01.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:01.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ca10 con 0x7f63d8005e00 2021-03-09T10:46:01.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:01.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:01.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19432 cs=0 l=1 rev1=1 rx=0x7f63f004c830 tx=0x7f63f007a330).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:01.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:46:01.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.947+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:01.960 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:01.952+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54166 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:01.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:46:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.954+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.954+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.954+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:01.962 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:01.963 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:01.963 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:01.963 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:01.963 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:01.963 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:01.964 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:01.964 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:01.964 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:01.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8005e00 2021-03-09T10:46:01.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.956+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:01.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.956+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:46:01.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:01.956+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8005e00 2021-03-09T10:46:02.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:02.740+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:02.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:02.740+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:03.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:03.741+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:03.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:03.741+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:04.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:04.740+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:04.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:04.740+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:05.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:05.708+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:05.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:05.741+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:05.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:05.741+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:06.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.740+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:06.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.740+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:06.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:06.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:06.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19432 cs=0 l=1 rev1=1 rx=0x7f63f004c830 tx=0x7f63f007a330).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:06.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19432 cs=0 l=1 rev1=1 rx=0x7f63f004c830 tx=0x7f63f007a330).stop 2021-03-09T10:46:06.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:06.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:06.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:06.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:06.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:46:06.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:06.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:06.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:06.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c140 con 0x7f63d8003400 2021-03-09T10:46:06.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:06.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:46:06.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:06.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b400 con 0x7f63d8003840 2021-03-09T10:46:06.959 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:06.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:06.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:06.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:06.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:06.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:06.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:46:06.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:06.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:06.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:06.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:06.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.953+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:06.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:06.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:06.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:06.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:06.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:06.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f63d8003840 2021-03-09T10:46:06.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:06.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:06.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079dc0 con 0x7f63d8003840 2021-03-09T10:46:06.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:06.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.954+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:06.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.955+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:46:06.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.955+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:06.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.955+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a130 secure :-1 s=READY pgs=19445 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f007ee60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:06.968 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:06.959+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54206 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:06.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:46:06.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.960+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:06.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.960+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:06.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.960+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:06.969 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:06.969 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:06.970 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:06.970 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:06.970 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:06.970 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:06.970 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:06.971 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:06.971 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:06.971 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:06.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:46:06.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.960+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:06.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.960+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:46:06.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:06.962+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003e180 con 0x7f63d8003840 2021-03-09T10:46:07.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:07.741+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:07.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:07.741+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:08.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:08.741+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:08.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:08.741+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:09.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:09.741+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:09.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:09.741+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:10.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:10.709+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:10.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:10.742+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:10.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:10.742+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:10.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:10.742+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:11.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.742+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:11.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.742+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:11.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:11.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:11.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a130 secure :-1 s=READY pgs=19445 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f007ee60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:11.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a130 secure :-1 s=READY pgs=19445 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f007ee60).stop 2021-03-09T10:46:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002a130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:11.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002a130 unknown :-1 s=CLOSED pgs=19445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:11.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:46:11.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:11.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:11.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:11.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:11.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:11.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:11.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:11.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:46:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006c7b0 con 0x7f63d8009130 2021-03-09T10:46:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:11.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:46:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:11.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:11.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.961+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.961+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:46:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.961+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.961+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.962+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:11.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.962+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:11.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.962+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:11.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:11.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:11.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:11.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:11.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:11.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:11.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:11.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:11.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:11.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0023030 con 0x7f63d8005e00 2021-03-09T10:46:11.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:11.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:11.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f63d8005e00 2021-03-09T10:46:11.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:11.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.963+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:11.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.964+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:46:11.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.964+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19466 cs=0 l=1 rev1=1 rx=0x7f63f0060010 tx=0x7f63f000dee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:11.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.964+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:11.976 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:11.967+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54245 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:11.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:46:11.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.969+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:11.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.969+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:11.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.969+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:11.978 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:11.978 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:11.978 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:11.978 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:11.978 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:11.979 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:11.979 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:11.979 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:11.979 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:11.979 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:46:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.970+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.970+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:46:11.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:11.971+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003e180 con 0x7f63d8005e00 2021-03-09T10:46:12.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:12.742+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:12.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:12.742+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:13.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:13.741+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:13.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:13.741+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:14.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:14.742+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:14.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:14.742+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:15.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:15.710+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:15.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:15.742+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:15.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:15.742+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:16.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.742+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:16.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.742+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:16.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:16.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:16.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19466 cs=0 l=1 rev1=1 rx=0x7f63f0060010 tx=0x7f63f000dee0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:16.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19466 cs=0 l=1 rev1=1 rx=0x7f63f0060010 tx=0x7f63f000dee0).stop 2021-03-09T10:46:16.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:16.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:16.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:16.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:16.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:46:16.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:16.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:16.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:16.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:16.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:16.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:16.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:16.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266f0 con 0x7f63d8003400 2021-03-09T10:46:16.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:16.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011db0 con 0x7f63d8009130 2021-03-09T10:46:16.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:16.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025110 con 0x7f63d8003840 2021-03-09T10:46:16.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.968+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:16.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.969+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:16.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.969+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:16.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.969+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:16.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.969+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:16.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.969+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:16.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.969+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:46:16.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.969+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:16.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.969+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:16.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:16.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:16.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:16.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.970+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:16.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:16.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:16.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:16.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:16.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:16.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:16.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:16.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:16.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:46:16.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:16.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:16.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0023030 con 0x7f63d8003840 2021-03-09T10:46:16.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:16.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:16.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=19477 cs=0 l=1 rev1=1 rx=0x7f63f0048590 tx=0x7f63f0088ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:16.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:46:16.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.971+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:16.984 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:16.975+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54285 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:16.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0048600 con 0x7f63d8003840 2021-03-09T10:46:16.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.976+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:16.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.976+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:16.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.976+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:16.986 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:16.986 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:16.987 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:16.987 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:16.987 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:16.987 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:16.987 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:16.988 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:16.988 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:16.988 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:16.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00483d0 con 0x7f63d8003840 2021-03-09T10:46:16.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.978+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:16.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.978+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:46:16.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:16.979+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f003e180 con 0x7f63d8003840 2021-03-09T10:46:17.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:17.743+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:17.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:17.743+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:18.743+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:18.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:18.743+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:19.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:19.743+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:19.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:19.743+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:20.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:20.711+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:20.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:20.742+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:20.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:20.742+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:20.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:20.742+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:21.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.743+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:21.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.743+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:21.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:21.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:21.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=19477 cs=0 l=1 rev1=1 rx=0x7f63f0048590 tx=0x7f63f0088ff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:21.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=19477 cs=0 l=1 rev1=1 rx=0x7f63f0048590 tx=0x7f63f0088ff0).stop 2021-03-09T10:46:21.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:21.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:21.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=CLOSED pgs=19477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:46:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:21.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00154a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:21.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:21.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:21.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:21.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:21.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:46:21.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:21.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00684b0 con 0x7f63d8009130 2021-03-09T10:46:21.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:21.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068830 con 0x7f63d8005e00 2021-03-09T10:46:21.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:21.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:21.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:21.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:21.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:21.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00154a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:21.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:46:21.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:21.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.976+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:21.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.977+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:21.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.977+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:21.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.977+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:21.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:21.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:21.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:21.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:21.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:21.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:46:21.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:46:21.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:21.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:21.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002fb80 con 0x7f63d8005e00 2021-03-09T10:46:21.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:21.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:21.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0020ef0 con 0x7f63d8005e00 2021-03-09T10:46:21.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:46:21.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.978+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:21.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.979+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00154a0 secure :-1 s=READY pgs=19490 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc0488e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:21.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.979+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:46:21.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.979+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:21.992 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:21.983+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54324 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:21.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc05d7a0 con 0x7f63d8005e00 2021-03-09T10:46:21.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.984+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:21.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.984+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:21.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.984+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:21.993 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:21.994 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:21.994 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:21.994 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:21.994 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:21.995 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:21.995 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:21.995 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:21.995 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:21.995 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:21.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05d7a0 con 0x7f63d8005e00 2021-03-09T10:46:21.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.987+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:21.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.987+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:46:21.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:21.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012cb0 con 0x7f63d8005e00 2021-03-09T10:46:22.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:22.742+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:22.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:22.742+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:23.743+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:23.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:23.743+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:24.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:24.743+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:24.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:24.743+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:25.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:25.712+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:25.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:25.744+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:25.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:25.744+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:26.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.744+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:26.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.744+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:26.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00154a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:26.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00154a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:26.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00154a0 secure :-1 s=READY pgs=19490 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc0488e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:26.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00154a0 secure :-1 s=READY pgs=19490 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc0488e0).stop 2021-03-09T10:46:26.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:26.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:26.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00154a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:26.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00154a0 unknown :-1 s=CLOSED pgs=19490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:26.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:46:26.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:26.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:26.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:26.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:26.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:26.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:26.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:26.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c140 con 0x7f63d8003400 2021-03-09T10:46:26.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:26.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:46:26.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:26.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a350 con 0x7f63d8003840 2021-03-09T10:46:26.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:26.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:26.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:26.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:26.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:26.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:26.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:46:26.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:26.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.984+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:26.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.985+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:26.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.985+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:26.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.985+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:26.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:26.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:26.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067d20 con 0x7f63d8003840 2021-03-09T10:46:26.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:46:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.986+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:26.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=19507 cs=0 l=1 rev1=1 rx=0x7f63f00294a0 tx=0x7f63f0009ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:27.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.987+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:46:27.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.987+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:27.000 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:26.992+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54364 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:27.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:46:27.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.993+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:27.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.993+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:27.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.993+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:27.002 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:27.003 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:27.004 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:27.004 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:27.004 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:27.004 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:27.005 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:27.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006610 con 0x7f63d8003840 2021-03-09T10:46:27.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.993+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:27.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.993+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:46:27.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:26.994+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63d8003840 2021-03-09T10:46:27.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:27.743+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:27.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:27.743+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:28.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:28.744+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:28.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:28.744+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:29.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:29.744+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:29.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:29.744+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:30.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:30.713+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:30.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:30.744+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:30.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:30.744+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:30.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:30.744+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:31.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.744+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:31.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.744+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:31.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:31.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:31.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=19507 cs=0 l=1 rev1=1 rx=0x7f63f00294a0 tx=0x7f63f0009ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:31.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=19507 cs=0 l=1 rev1=1 rx=0x7f63f00294a0 tx=0x7f63f0009ec0).stop 2021-03-09T10:46:31.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:31.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:31.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:31.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=CLOSED pgs=19507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:31.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:46:31.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:31.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:31.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:31.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:31.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:31.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:31.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:31.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b400 con 0x7f63d8003400 2021-03-09T10:46:32.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:32.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8009130 2021-03-09T10:46:32.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:32.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:46:32.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:32.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:32.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:32.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:32.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:32.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:32.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:46:32.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:32.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:32.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:32.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:32.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.993+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:32.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:32.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:32.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:32.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:32.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:32.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8005e00 2021-03-09T10:46:32.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:32.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:32.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067d20 con 0x7f63d8005e00 2021-03-09T10:46:32.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:32.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:32.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19514 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63f0012520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:32.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.994+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:46:32.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:31.995+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:32.007 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:32.000+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54403 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:32.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8005e00 2021-03-09T10:46:32.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.000+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:32.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.000+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:32.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.000+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:32.009 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:32.009 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:32.010 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:32.010 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:32.010 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:32.010 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:32.011 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:32.011 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:32.011 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:32.011 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:32.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.000+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:46:32.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.000+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:32.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.000+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:46:32.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63d8005e00 2021-03-09T10:46:32.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.745+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:32.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:32.745+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:33.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:33.745+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:33.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:33.745+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:34.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:34.745+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:34.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:34.745+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:35.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:35.714+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:35.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:35.745+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:35.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:35.745+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:36.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:36.745+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:36.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:36.745+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:37.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:46:37.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:37.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19514 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63f0012520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:37.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19514 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63f0012520).stop 2021-03-09T10:46:37.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:37.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:46:37.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:37.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:37.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:37.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:37.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:37.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:37.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:37.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00684b0 con 0x7f63d8003400 2021-03-09T10:46:37.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:37.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:46:37.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:37.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ce0 con 0x7f63d8003840 2021-03-09T10:46:37.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:37.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:37.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:37.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:37.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.001+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:37.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.002+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:37.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.002+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:46:37.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.002+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:37.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.002+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:37.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.002+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:37.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.002+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:37.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.002+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:37.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:37.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:37.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0020ef0 con 0x7f63d8003840 2021-03-09T10:46:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:37.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:46:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.003+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.004+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:46:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.004+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=19532 cs=0 l=1 rev1=1 rx=0x7f63f0031e80 tx=0x7f63f0028310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:37.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.004+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:37.017 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:37.008+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54443 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:37.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.008+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:46:37.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.008+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:37.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.008+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:37.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.008+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:37.019 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:37.019 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:37.019 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:37.020 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:37.020 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:37.020 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:37.021 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:37.021 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:37.021 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:37.021 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:37.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.011+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:46:37.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.011+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:37.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.011+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:46:37.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.012+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f001d330 con 0x7f63d8003840 2021-03-09T10:46:37.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.745+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:37.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:37.745+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:38.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:38.745+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:38.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:38.745+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:39.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:39.746+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:39.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:39.746+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:40.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:40.715+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:40.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:40.746+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:40.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:40.746+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:40.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:40.746+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:41.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:41.746+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:41.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:41.746+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:42.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:46:42.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:42.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=19532 cs=0 l=1 rev1=1 rx=0x7f63f0031e80 tx=0x7f63f0028310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:42.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=19532 cs=0 l=1 rev1=1 rx=0x7f63f0031e80 tx=0x7f63f0028310).stop 2021-03-09T10:46:42.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:42.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:42.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:42.013 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=CLOSED pgs=19532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:42.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:46:42.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:42.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:42.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:42.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:42.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:42.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:42.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:42.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:46:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067d70 con 0x7f63d8009130 2021-03-09T10:46:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0030500 con 0x7f63d8005e00 2021-03-09T10:46:42.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:42.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:42.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:42.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:42.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.009+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:42.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.010+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:42.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.011+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:46:42.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.011+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:42.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.011+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:42.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.011+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:42.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.011+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:42.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.011+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:42.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:42.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:42.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:42.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:42.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:42.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:42.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:42.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:42.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:42.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e110 con 0x7f63d8005e00 2021-03-09T10:46:42.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0020ef0 con 0x7f63d8005e00 2021-03-09T10:46:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:42.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19553 cs=0 l=1 rev1=1 rx=0x7f63f001dff0 tx=0x7f63f001d620).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:42.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:46:42.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.012+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:42.024 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:42.015+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54482 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:42.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f4c0 con 0x7f63d8005e00 2021-03-09T10:46:42.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.016+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:42.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.016+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:42.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.016+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:42.026 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:42.026 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:42.026 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:42.027 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:42.027 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:42.027 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:42.027 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:42.027 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:42.028 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:42.028 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:42.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.020+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63d8005e00 2021-03-09T10:46:42.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.020+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:42.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.020+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:46:42.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8005e00 2021-03-09T10:46:42.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.746+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:42.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:42.746+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:43.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:43.746+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:43.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:43.746+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:44.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:44.746+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:44.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:44.746+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:45.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:45.716+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:45.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:45.746+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:45.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:45.746+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:46.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:46.747+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:46.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:46.747+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:47.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:47.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:47.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19553 cs=0 l=1 rev1=1 rx=0x7f63f001dff0 tx=0x7f63f001d620).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19553 cs=0 l=1 rev1=1 rx=0x7f63f001dff0 tx=0x7f63f001d620).stop 2021-03-09T10:46:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:47.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:46:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:47.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:47.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:47.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:47.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d7f0 con 0x7f63d8003400 2021-03-09T10:46:47.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:47.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:46:47.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ce0 con 0x7f63d8003840 2021-03-09T10:46:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:47.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:47.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.016+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:47.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.017+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:47.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.017+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:46:47.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.017+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:47.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.017+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:47.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.017+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:47.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.017+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:47.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.017+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:47.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:47.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:47.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:47.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:47.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:47.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:46:47.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:46:47.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:47.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:47.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d430 con 0x7f63d8003840 2021-03-09T10:46:47.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:47.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:47.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002fb80 con 0x7f63d8003840 2021-03-09T10:46:47.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:46:47.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.018+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:47.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.020+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:46:47.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.020+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:47.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.020+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=19560 cs=0 l=1 rev1=1 rx=0x7f63e0017a80 tx=0x7f63fc01b1d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:47.032 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:47.023+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54522 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:47.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fda0 con 0x7f63d8003840 2021-03-09T10:46:47.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.025+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:47.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.025+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:47.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.025+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:47.034 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:47.034 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:47.034 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:47.034 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:47.035 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:47.035 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:47.035 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:47.035 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:47.035 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:47.036 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:47.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03ff00 con 0x7f63d8003840 2021-03-09T10:46:47.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.025+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:47.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8003840 2021-03-09T10:46:47.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.025+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:46:47.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.747+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:47.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:47.747+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:48.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:48.747+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:48.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:48.747+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:49.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:49.747+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:49.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:49.747+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:50.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:50.717+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:50.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:50.747+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:50.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:50.747+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:50.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:50.747+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:51.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:51.747+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:51.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:51.747+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.024+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.024+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:52.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.024+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=19560 cs=0 l=1 rev1=1 rx=0x7f63e0017a80 tx=0x7f63fc01b1d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:52.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.024+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 secure :-1 s=READY pgs=19560 cs=0 l=1 rev1=1 rx=0x7f63e0017a80 tx=0x7f63fc01b1d0).stop 2021-03-09T10:46:52.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:52.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:52.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002da40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:52.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002da40 unknown :-1 s=CLOSED pgs=19560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:52.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:46:52.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:52.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:52.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:52.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:52.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:52.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:52.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:52.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b990 con 0x7f63d8003400 2021-03-09T10:46:52.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:52.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c140 con 0x7f63d8009130 2021-03-09T10:46:52.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:52.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8005e00 2021-03-09T10:46:52.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:52.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:52.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:52.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:52.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:52.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.025+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:52.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:46:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:52.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:52.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:52.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:52.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:52.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:52.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:52.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:52.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:52.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:52.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0018200 con 0x7f63d8005e00 2021-03-09T10:46:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e110 con 0x7f63d8005e00 2021-03-09T10:46:52.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:52.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.026+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:52.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.027+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19572 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f00553c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:52.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.028+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:46:52.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.028+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:52.039 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:52.032+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54561 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8005e00 2021-03-09T10:46:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.032+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.032+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.032+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:52.040 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:52.041 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:52.041 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:52.041 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:52.041 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:52.042 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:52.042 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:52.042 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:52.042 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:52.042 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:52.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.034+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f005dc70 con 0x7f63d8005e00 2021-03-09T10:46:52.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.034+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:52.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.034+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:46:52.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.035+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8005e00 2021-03-09T10:46:52.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.747+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:52.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:52.747+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:53.748+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:53.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:53.748+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:54.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:54.748+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:54.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:54.748+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:55.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:55.718+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:46:55.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:55.748+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:55.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:55.748+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:56.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:56.748+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:56.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:56.748+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:57.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:46:57.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:46:57.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19572 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f00553c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:46:57.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 secure :-1 s=READY pgs=19572 cs=0 l=1 rev1=1 rx=0x7f63f0006b70 tx=0x7f63f00553c0).stop 2021-03-09T10:46:57.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:57.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:46:57.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00798d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:46:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00798d0 unknown :-1 s=CLOSED pgs=19572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:46:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:46:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:46:57.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:57.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:46:57.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:46:57.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:46:57.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:57.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007af60 con 0x7f63d8003400 2021-03-09T10:46:57.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:57.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8009130 2021-03-09T10:46:57.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:46:57.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f63d8003840 2021-03-09T10:46:57.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:46:57.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:46:57.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:57.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:46:57.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:46:57.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:46:57.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.034+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:46:57.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.034+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:46:57.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.034+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:46:57.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:46:57.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:46:57.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:46:57.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:46:57.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:57.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:57.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:46:57.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:46:57.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:46:57.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:46:57.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:46:57.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:57.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0020ef0 con 0x7f63d8003840 2021-03-09T10:46:57.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:46:57.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:46:57.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0018200 con 0x7f63d8003840 2021-03-09T10:46:57.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:46:57.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.035+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:57.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.036+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e090 secure :-1 s=READY pgs=19586 cs=0 l=1 rev1=1 rx=0x7f63f007a820 tx=0x7f63f007e490).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:46:57.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.036+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:46:57.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.036+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:46:57.047 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:46:57.040+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54601 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:46:57.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007a430 con 0x7f63d8003840 2021-03-09T10:46:57.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.041+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:46:57.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.041+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:46:57.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.041+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:46:57.048 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:46:57.048 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:46:57.049 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:57.049 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:46:57.049 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:46:57.049 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:46:57.050 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:46:57.050 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:46:57.050 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:46:57.050 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:46:57.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007a250 con 0x7f63d8003840 2021-03-09T10:46:57.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.041+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:46:57.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.041+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:46:57.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8003840 2021-03-09T10:46:57.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.748+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:57.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:57.748+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:58.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:58.748+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:58.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:58.748+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:46:59.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:59.748+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:46:59.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:46:59.748+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:00.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:00.719+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:00.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:00.748+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:00.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:00.748+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:00.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:00.748+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:01.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:01.749+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:01.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:01.749+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:02.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.040+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:02.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.040+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.040+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e090 secure :-1 s=READY pgs=19586 cs=0 l=1 rev1=1 rx=0x7f63f007a820 tx=0x7f63f007e490).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.040+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e090 secure :-1 s=READY pgs=19586 cs=0 l=1 rev1=1 rx=0x7f63f007a820 tx=0x7f63f007e490).stop 2021-03-09T10:47:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.040+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:02.043 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e005e090 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e005e090 unknown :-1 s=CLOSED pgs=19586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:47:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:02.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:02.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:02.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:02.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:02.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:02.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d7f0 con 0x7f63d8003400 2021-03-09T10:47:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ce0 con 0x7f63d8009130 2021-03-09T10:47:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:02.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8005e00 2021-03-09T10:47:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:02.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:47:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:02.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:02.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:02.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:02.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.041+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:02.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:02.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:02.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:02.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:02.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:02.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:47:02.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:47:02.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:02.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:02.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006e110 con 0x7f63d8005e00 2021-03-09T10:47:02.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:02.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:02.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0020ef0 con 0x7f63d8005e00 2021-03-09T10:47:02.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:47:02.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.042+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:02.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.043+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:47:02.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.043+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:02.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079880 secure :-1 s=READY pgs=19597 cs=0 l=1 rev1=1 rx=0x7f63e002f2d0 tx=0x7f63f0040640).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:02.059 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:02.049+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54640 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:02.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8005e00 2021-03-09T10:47:02.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.049+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:02.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.049+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:02.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.049+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:02.060 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:02.061 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:02.061 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:02.061 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:02.061 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:02.061 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:02.062 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:02.062 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:02.062 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:02.062 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:02.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.049+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8005e00 2021-03-09T10:47:02.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.049+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:02.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.049+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:47:02.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8005e00 2021-03-09T10:47:02.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.749+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:02.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:02.749+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:03.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:03.749+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:03.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:03.749+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:04.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:04.749+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:04.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:04.749+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:05.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:05.720+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:05.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:05.749+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:05.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:05.749+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:06.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:06.749+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:06.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:06.749+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:07.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.049+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:07.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.049+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:07.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.049+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079880 secure :-1 s=READY pgs=19597 cs=0 l=1 rev1=1 rx=0x7f63e002f2d0 tx=0x7f63f0040640).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:07.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.049+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079880 secure :-1 s=READY pgs=19597 cs=0 l=1 rev1=1 rx=0x7f63e002f2d0 tx=0x7f63f0040640).stop 2021-03-09T10:47:07.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:07.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:07.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079880 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:07.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079880 unknown :-1 s=CLOSED pgs=19597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:07.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:47:07.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:07.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:07.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:07.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:07.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:07.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:07.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:07.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012c70 con 0x7f63d8003400 2021-03-09T10:47:07.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:07.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b990 con 0x7f63d8009130 2021-03-09T10:47:07.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:07.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014ec0 con 0x7f63d8003840 2021-03-09T10:47:07.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:07.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:07.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:07.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:07.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.050+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:07.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:07.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.051+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:47:07.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.051+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:07.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.051+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:07.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:07.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:07.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.051+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:07.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:07.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:07.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:07.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:07.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:07.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:47:07.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:47:07.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:07.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:07.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f63d8003840 2021-03-09T10:47:07.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:07.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:07.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006e110 con 0x7f63d8003840 2021-03-09T10:47:07.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:47:07.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:07.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e40 secure :-1 s=READY pgs=19612 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0013bf0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:07.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:47:07.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.052+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:07.064 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:07.057+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54680 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:07.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:47:07.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.057+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:07.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.057+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:07.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.057+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:07.066 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:07.066 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:07.066 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:07.066 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:07.066 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:07.067 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:07.067 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:07.067 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:07.067 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:07.068 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:07.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:47:07.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.057+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:07.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.057+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:47:07.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8003840 2021-03-09T10:47:07.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.749+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:07.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:07.749+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:08.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:08.749+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:08.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:08.749+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:09.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:09.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:09.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:09.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:10.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:10.721+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:10.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:10.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:10.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:10.750+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:11.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:11.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:11.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:11.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:12.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:12.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:12.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e40 secure :-1 s=READY pgs=19612 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0013bf0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:12.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e40 secure :-1 s=READY pgs=19612 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0013bf0).stop 2021-03-09T10:47:12.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:12.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:12.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029e40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:12.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029e40 unknown :-1 s=CLOSED pgs=19612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:12.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:47:12.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:12.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:12.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:12.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:12.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:12.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:12.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:12.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b600 con 0x7f63d8003400 2021-03-09T10:47:12.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:12.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:47:12.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:12.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029ce0 con 0x7f63d8005e00 2021-03-09T10:47:12.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.057+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:12.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:12.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:12.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:12.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:12.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:12.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:47:12.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:12.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:12.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:12.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.058+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:12.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:12.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:12.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:12.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:12.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:12.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:12.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:47:12.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:47:12.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:12.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:12.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013d00 con 0x7f63d8005e00 2021-03-09T10:47:12.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:12.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:12.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d430 con 0x7f63d8005e00 2021-03-09T10:47:12.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:47:12.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.059+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:12.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.060+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19628 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04bd00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:12.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.060+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:47:12.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.060+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:12.072 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:12.064+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54719 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:12.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.065+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03fcd0 con 0x7f63d8005e00 2021-03-09T10:47:12.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.065+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:12.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.065+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:12.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.065+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:12.074 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:12.074 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:12.075 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:12.075 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:12.075 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:12.075 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:12.075 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:12.076 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:12.076 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:12.076 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:12.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03fcd0 con 0x7f63d8005e00 2021-03-09T10:47:12.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.068+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:12.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.068+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:47:12.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.068+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63d8005e00 2021-03-09T10:47:12.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:12.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:12.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:13.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:13.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:13.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:13.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:14.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:14.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:14.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:14.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:15.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:15.721+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:15.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:15.749+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:15.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:15.749+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:16.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:16.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:16.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:16.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:17.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:17.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:17.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19628 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04bd00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:17.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19628 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04bd00).stop 2021-03-09T10:47:17.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:17.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:17.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:17.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=19628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:17.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:47:17.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:17.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:17.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:17.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:17.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:17.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:17.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:17.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:47:17.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:17.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:47:17.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:17.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266f0 con 0x7f63d8003840 2021-03-09T10:47:17.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:17.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:17.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:17.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:17.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:17.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:17.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:47:17.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:17.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.064+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:17.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.065+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:17.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.065+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:17.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.065+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:17.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:17.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:17.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:17.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:17.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:17.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:47:17.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:47:17.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:17.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:17.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d430 con 0x7f63d8003840 2021-03-09T10:47:17.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:17.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:17.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013d00 con 0x7f63d8003840 2021-03-09T10:47:17.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:47:17.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.066+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:17.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.067+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 secure :-1 s=READY pgs=19643 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc065890).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:17.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.067+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:47:17.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.067+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:17.079 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:17.072+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54759 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:17.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.073+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047530 con 0x7f63d8003840 2021-03-09T10:47:17.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.073+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:17.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.073+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:17.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.073+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:17.081 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:17.081 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:17.081 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:17.082 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:17.082 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:17.082 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:17.082 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:17.083 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:17.083 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:17.083 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:17.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.076+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047530 con 0x7f63d8003840 2021-03-09T10:47:17.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.076+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:17.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.076+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:47:17.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc052b40 con 0x7f63d8003840 2021-03-09T10:47:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:17.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:17.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:18.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:18.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:18.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:18.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:19.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:19.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:19.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:19.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:20.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:20.722+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:20.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:20.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:20.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:20.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:20.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:20.750+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:21.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:21.750+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:21.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:21.750+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:22.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:22.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:22.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 secure :-1 s=READY pgs=19643 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc065890).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:22.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 secure :-1 s=READY pgs=19643 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc065890).stop 2021-03-09T10:47:22.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:22.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:22.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:22.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=19643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:22.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:47:22.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:22.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:22.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:22.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:22.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:22.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:22.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:22.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c390 con 0x7f63d8003400 2021-03-09T10:47:22.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:22.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f63d8009130 2021-03-09T10:47:22.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:22.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8005e00 2021-03-09T10:47:22.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:22.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:22.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.072+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:22.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.073+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:22.080 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.073+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:22.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.073+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:22.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.073+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:47:22.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.073+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:22.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.073+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:22.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:22.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:22.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.073+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:22.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:22.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:22.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:47:22.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:47:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b940 con 0x7f63d8005e00 2021-03-09T10:47:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:22.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f63d8005e00 2021-03-09T10:47:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:47:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.074+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079eb0 secure :-1 s=READY pgs=19654 cs=0 l=1 rev1=1 rx=0x7f63f0031e50 tx=0x7f63f000df50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:22.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.075+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:47:22.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.075+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:22.087 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:22.080+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54798 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:22.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8005e00 2021-03-09T10:47:22.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.081+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:22.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.081+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:22.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.081+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:22.089 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:22.090 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:22.091 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:22.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:47:22.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.081+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:22.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.081+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:47:22.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.081+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8005e00 2021-03-09T10:47:22.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.751+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:22.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:22.751+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:23.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:23.751+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:23.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:23.751+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:24.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:24.751+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:24.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:24.751+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:25.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:25.723+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:25.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:25.751+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:25.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:25.751+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:26.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:26.751+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:26.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:26.751+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:27.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:27.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:27.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079eb0 secure :-1 s=READY pgs=19654 cs=0 l=1 rev1=1 rx=0x7f63f0031e50 tx=0x7f63f000df50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:27.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079eb0 secure :-1 s=READY pgs=19654 cs=0 l=1 rev1=1 rx=0x7f63f0031e50 tx=0x7f63f000df50).stop 2021-03-09T10:47:27.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:27.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:27.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0079eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:27.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0079eb0 unknown :-1 s=CLOSED pgs=19654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:27.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:47:27.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:27.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:27.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:27.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:27.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:27.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:27.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:27.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028c60 con 0x7f63d8003400 2021-03-09T10:47:27.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:27.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001470 con 0x7f63d8009130 2021-03-09T10:47:27.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:27.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b600 con 0x7f63d8003840 2021-03-09T10:47:27.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:27.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:27.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.080+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:27.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.081+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:27.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:27.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.081+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:27.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.081+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:47:27.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.081+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:27.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.081+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:27.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.081+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:27.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:27.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:27.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:27.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:27.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:27.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:47:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:47:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:27.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:27.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013d00 con 0x7f63d8003840 2021-03-09T10:47:27.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:27.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:27.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b940 con 0x7f63d8003840 2021-03-09T10:47:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:47:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.082+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.083+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19669 cs=0 l=1 rev1=1 rx=0x7f63f0028fa0 tx=0x7f63f0088350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.083+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:47:27.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.083+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:27.096 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:27.087+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54838 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:27.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004d110 con 0x7f63d8003840 2021-03-09T10:47:27.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.088+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:27.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.088+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:27.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.088+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:27.097 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:27.098 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:27.098 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:27.098 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:27.098 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:27.099 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:27.099 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.094+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:47:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.095+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8003840 2021-03-09T10:47:27.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.095+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:47:27.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.751+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:27.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:27.751+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:28.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:28.751+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:28.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:28.751+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:29.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:29.752+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:29.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:29.752+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:30.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:30.724+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:30.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:30.752+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:30.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:30.752+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:30.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:30.752+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:31.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:31.752+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:31.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:31.752+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:32.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:32.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:32.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19669 cs=0 l=1 rev1=1 rx=0x7f63f0028fa0 tx=0x7f63f0088350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:32.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19669 cs=0 l=1 rev1=1 rx=0x7f63f0028fa0 tx=0x7f63f0088350).stop 2021-03-09T10:47:32.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:32.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:32.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:32.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=CLOSED pgs=19669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:32.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:47:32.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00652f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:32.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:32.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:32.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:32.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:32.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:32.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:32.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T10:47:32.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:32.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bf70 con 0x7f63d8009130 2021-03-09T10:47:32.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:32.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d830 con 0x7f63d8005e00 2021-03-09T10:47:32.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:32.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:32.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:32.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:32.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:47:32.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:32.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:32.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:32.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:32.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:32.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:32.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.088+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:32.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:32.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:47:32.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:47:32.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:32.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:32.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8005e00 2021-03-09T10:47:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013d00 con 0x7f63d8005e00 2021-03-09T10:47:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:47:32.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.089+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:32.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.090+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00652f0 secure :-1 s=READY pgs=19682 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0055380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:32.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.090+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:47:32.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.090+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:32.102 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:32.094+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54886 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:32.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.096+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:47:32.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.096+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:32.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.096+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:32.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.096+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:32.104 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:32.104 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:32.105 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:32.105 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:32.105 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:32.105 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:32.106 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:32.106 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:32.106 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:32.106 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:32.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8005e00 2021-03-09T10:47:32.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.100+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:32.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.100+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:47:32.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.101+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00883c0 con 0x7f63d8005e00 2021-03-09T10:47:32.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.752+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:32.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:32.752+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:33.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:33.752+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:33.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:33.752+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:34.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:34.752+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:34.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:34.752+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:35.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:35.725+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:35.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:35.752+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:35.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:35.752+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:36.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:36.752+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:36.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:36.752+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:37.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00652f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00652f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00652f0 secure :-1 s=READY pgs=19682 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0055380).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00652f0 secure :-1 s=READY pgs=19682 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0055380).stop 2021-03-09T10:47:37.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:37.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:37.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e00652f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:37.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e00652f0 unknown :-1 s=CLOSED pgs=19682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:37.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:47:37.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:37.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:37.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:37.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:37.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003400 2021-03-09T10:47:37.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b600 con 0x7f63d8009130 2021-03-09T10:47:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:47:37.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.095+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:37.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:37.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:47:37.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:37.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:37.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.096+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:37.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:37.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:37.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:37.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:37.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:47:37.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:47:37.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:37.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:37.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e760 con 0x7f63d8003840 2021-03-09T10:47:37.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:37.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:37.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d430 con 0x7f63d8003840 2021-03-09T10:47:37.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:47:37.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.097+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.098+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19696 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.098+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:47:37.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.098+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:37.111 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:37.102+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54936 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:37.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.104+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047530 con 0x7f63d8003840 2021-03-09T10:47:37.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.104+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:37.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.105+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:37.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.105+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:37.112 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:37.113 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:37.113 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:37.113 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:37.113 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:37.114 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:37.114 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:37.114 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:37.114 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:37.114 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.107+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02ce20 con 0x7f63d8003840 2021-03-09T10:47:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.107+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8003840 2021-03-09T10:47:37.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.108+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:47:37.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.753+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:37.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:37.753+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:38.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:38.753+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:38.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:38.753+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:39.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:39.753+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:39.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:39.753+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:40.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:40.726+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:40.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:40.753+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:40.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:40.753+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:40.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:40.753+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:41.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:41.753+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:41.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:41.753+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:42.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:42.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:42.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19696 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:42.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19696 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc009990).stop 2021-03-09T10:47:42.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=CLOSED pgs=19696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:47:42.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:42.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T10:47:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025150 con 0x7f63d8009130 2021-03-09T10:47:42.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:42.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8005e00 2021-03-09T10:47:42.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:42.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:42.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.103+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:42.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:42.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:42.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:42.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:47:42.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:42.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:42.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:42.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:42.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:42.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.104+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:42.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:42.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:42.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:42.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:42.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:47:42.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:47:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007a3a0 con 0x7f63d8005e00 2021-03-09T10:47:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:42.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:42.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8005e00 2021-03-09T10:47:42.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:47:42.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:42.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19709 cs=0 l=1 rev1=1 rx=0x7f63f0050010 tx=0x7f63f0039e60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:42.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:47:42.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.105+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:42.118 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:42.110+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 54973 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:42.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8005e00 2021-03-09T10:47:42.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.110+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:42.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.110+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:42.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.110+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:42.120 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:42.120 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:42.121 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:42.121 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:42.121 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:42.121 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:42.121 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:42.122 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:42.122 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:42.122 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:42.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8005e00 2021-03-09T10:47:42.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.110+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:42.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.110+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:47:42.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.111+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8005e00 2021-03-09T10:47:42.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.753+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:42.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:42.753+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:43.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:43.753+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:43.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:43.753+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:44.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:44.754+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:44.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:44.754+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:45.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:45.727+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:45.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:45.754+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:45.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:45.754+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:46.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:46.754+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:46.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:46.754+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:47.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:47:47.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:47.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19709 cs=0 l=1 rev1=1 rx=0x7f63f0050010 tx=0x7f63f0039e60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:47.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19709 cs=0 l=1 rev1=1 rx=0x7f63f0050010 tx=0x7f63f0039e60).stop 2021-03-09T10:47:47.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:47.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:47.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:47.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=19709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:47.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:47:47.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:47.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:47.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:47.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:47.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:47.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:47.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:47.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003400 2021-03-09T10:47:47.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:47.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069640 con 0x7f63d8009130 2021-03-09T10:47:47.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:47.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:47:47.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.110+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.111+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.111+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:47.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.112+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.112+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:47:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.112+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.112+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.112+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:47.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.112+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:47.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.112+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:47.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:47.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:47.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:47:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:47:47.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:47.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:47.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068460 con 0x7f63d8003840 2021-03-09T10:47:47.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:47.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:47.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007a3a0 con 0x7f63d8003840 2021-03-09T10:47:47.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:47:47.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:47.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.113+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19725 cs=0 l=1 rev1=1 rx=0x7f63f0003590 tx=0x7f63f0042910).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:47.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.114+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:47:47.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.114+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:47.126 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:47.119+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55014 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:47.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0013b10 con 0x7f63d8003840 2021-03-09T10:47:47.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.119+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:47.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.119+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:47.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.119+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:47.128 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:47.128 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:47.128 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:47.128 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:47.128 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:47.129 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:47.129 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:47.129 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:47.129 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:47.129 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:47.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.120+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055470 con 0x7f63d8003840 2021-03-09T10:47:47.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.120+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:47.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.120+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:47:47.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.122+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005ddb0 con 0x7f63d8003840 2021-03-09T10:47:47.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.754+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:47.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:47.754+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:48.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:48.754+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:48.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:48.754+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:49.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:49.754+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:49.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:49.754+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:50.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:50.728+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:50.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:50.754+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:50.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:50.754+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:50.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:50.754+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:51.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:51.755+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:51.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:51.755+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19725 cs=0 l=1 rev1=1 rx=0x7f63f0003590 tx=0x7f63f0042910).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:52.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19725 cs=0 l=1 rev1=1 rx=0x7f63f0003590 tx=0x7f63f0042910).stop 2021-03-09T10:47:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:52.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=CLOSED pgs=19725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:47:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:52.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:52.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b600 con 0x7f63d8003400 2021-03-09T10:47:52.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:52.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:47:52.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:52.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce90 con 0x7f63d8005e00 2021-03-09T10:47:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.119+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.120+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:52.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.120+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:52.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.120+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:52.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.121+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:47:52.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.121+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:52.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.121+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:52.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.121+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:52.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.121+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:52.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.121+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:52.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:52.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:52.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:52.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:52.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:52.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:47:52.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:47:52.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:52.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:52.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8005e00 2021-03-09T10:47:52.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:52.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:52.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068460 con 0x7f63d8005e00 2021-03-09T10:47:52.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:47:52.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:52.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.122+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19742 cs=0 l=1 rev1=1 rx=0x7f63f0042ad0 tx=0x7f63f0042790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:52.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.123+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:47:52.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.123+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:52.135 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:52.126+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55054 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:52.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0040410 con 0x7f63d8005e00 2021-03-09T10:47:52.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.127+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:52.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.127+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:52.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.127+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:52.136 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:52.137 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:52.137 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:52.137 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:52.137 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:52.138 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:52.138 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:52.138 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:52.138 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:52.138 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:52.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.131+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0040230 con 0x7f63d8005e00 2021-03-09T10:47:52.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.131+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:52.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.131+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:47:52.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.132+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041730 con 0x7f63d8005e00 2021-03-09T10:47:52.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.755+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:52.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:52.755+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:53.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:53.754+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:53.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:53.754+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:54.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:54.755+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:54.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:54.755+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:55.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:55.729+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:47:55.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:55.755+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:55.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:55.755+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:56.755+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:56.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:56.755+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:57.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:47:57.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:47:57.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19742 cs=0 l=1 rev1=1 rx=0x7f63f0042ad0 tx=0x7f63f0042790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:47:57.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19742 cs=0 l=1 rev1=1 rx=0x7f63f0042ad0 tx=0x7f63f0042790).stop 2021-03-09T10:47:57.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:57.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:47:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:47:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=19742 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:47:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:47:57.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:57.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:47:57.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:57.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:47:57.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:47:57.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:47:57.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:57.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065090 con 0x7f63d8003400 2021-03-09T10:47:57.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:57.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068da0 con 0x7f63d8009130 2021-03-09T10:47:57.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:47:57.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003840 2021-03-09T10:47:57.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:47:57.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:47:57.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:57.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:47:57.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.127+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:47:57.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.128+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:47:57.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.128+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:47:57.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.128+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:47:57.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.128+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:47:57.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.128+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:47:57.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.128+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:47:57.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.128+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:47:57.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:47:57.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:57.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:57.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:47:57.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:47:57.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:47:57.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:47:57.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:47:57.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:57.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068460 con 0x7f63d8003840 2021-03-09T10:47:57.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:47:57.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:47:57.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e760 con 0x7f63d8003840 2021-03-09T10:47:57.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:47:57.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:57.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:47:57.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19752 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:47:57.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.129+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:47:57.145 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:47:57.134+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55092 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:47:57.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.135+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:47:57.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.135+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:47:57.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.135+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:47:57.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.135+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:47:57.147 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:47:57.147 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:47:57.147 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:57.148 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:47:57.148 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:47:57.148 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:47:57.148 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:47:57.149 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:47:57.149 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:47:57.149 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:47:57.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:47:57.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.137+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:47:57.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.137+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:47:57.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03fc70 con 0x7f63d8003840 2021-03-09T10:47:57.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.755+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:57.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:57.755+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:58.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:58.755+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:58.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:58.755+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:47:59.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:59.755+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:47:59.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:47:59.755+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:00.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:00.730+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:00.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:00.756+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:00.756+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:00.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:00.756+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:01.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:01.756+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:01.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:01.756+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:02.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:02.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:02.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19752 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:02.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19752 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a740).stop 2021-03-09T10:48:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:02.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=CLOSED pgs=19752 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:48:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:02.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:02.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:02.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:02.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:02.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:02.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001230 con 0x7f63d8003400 2021-03-09T10:48:02.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:02.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069640 con 0x7f63d8009130 2021-03-09T10:48:02.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017500 con 0x7f63d8005e00 2021-03-09T10:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:02.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:02.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.134+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:02.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.135+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:02.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.135+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.135+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.135+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.136+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.136+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:02.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.136+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:02.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:02.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:02.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:02.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:48:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:48:02.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e450 con 0x7f63d8005e00 2021-03-09T10:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:02.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068460 con 0x7f63d8005e00 2021-03-09T10:48:02.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:48:02.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.137+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:02.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.138+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:48:02.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.138+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:02.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.138+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19770 cs=0 l=1 rev1=1 rx=0x7f63e0079d80 tx=0x7f63fc01f420).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:02.150 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:02.141+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55133 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:02.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8005e00 2021-03-09T10:48:02.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.144+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.144+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.144+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:02.152 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:02.153 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:02.153 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:02.153 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:02.153 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:02.153 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:02.154 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:02.154 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:02.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.144+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0475e0 con 0x7f63d8005e00 2021-03-09T10:48:02.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.144+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:02.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.144+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:48:02.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.146+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047480 con 0x7f63d8005e00 2021-03-09T10:48:02.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.756+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:02.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:02.756+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:03.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:03.756+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:03.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:03.756+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:04.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:04.756+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:04.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:04.756+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:05.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:05.731+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:05.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:05.756+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:05.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:05.756+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:06.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:06.756+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:06.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:06.756+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:07.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:07.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:07.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19770 cs=0 l=1 rev1=1 rx=0x7f63e0079d80 tx=0x7f63fc01f420).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:07.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19770 cs=0 l=1 rev1=1 rx=0x7f63e0079d80 tx=0x7f63fc01f420).stop 2021-03-09T10:48:07.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:07.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:07.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:07.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=19770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:07.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:48:07.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:07.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:07.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:07.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:07.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:07.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:07.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:07.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:48:07.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:07.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8009130 2021-03-09T10:48:07.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:07.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:48:07.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:07.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:07.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:07.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:07.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.142+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:07.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.143+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:07.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.143+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:48:07.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.143+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:07.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.143+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:07.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.144+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:07.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.144+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:07.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.144+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:07.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:07.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:07.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:07.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:07.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:07.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:48:07.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:48:07.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:07.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:07.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016ed0 con 0x7f63d8003840 2021-03-09T10:48:07.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:07.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:07.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e450 con 0x7f63d8003840 2021-03-09T10:48:07.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:48:07.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:07.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19785 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc048950).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:07.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:48:07.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.145+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:07.158 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:07.149+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55171 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:07.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.152+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047550 con 0x7f63d8003840 2021-03-09T10:48:07.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.152+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:07.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.152+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:07.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.152+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:07.160 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:07.160 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:07.160 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:07.160 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:07.160 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:07.161 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:07.161 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:07.161 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:07.162 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:07.162 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:07.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047550 con 0x7f63d8003840 2021-03-09T10:48:07.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.156+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:07.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc066290 con 0x7f63d8003840 2021-03-09T10:48:07.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.156+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:48:07.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.757+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:07.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:07.757+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:08.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:08.757+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:08.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:08.757+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:09.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:09.756+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:09.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:09.756+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:10.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:10.732+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:10.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:10.757+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:10.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:10.757+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:10.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:10.757+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:11.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:11.757+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:11.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:11.757+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:12.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:48:12.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:12.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19785 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc048950).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:12.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 secure :-1 s=READY pgs=19785 cs=0 l=1 rev1=1 rx=0x7f63fc071660 tx=0x7f63fc048950).stop 2021-03-09T10:48:12.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:12.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:12.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0029cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:12.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0029cc0 unknown :-1 s=CLOSED pgs=19785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:12.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:48:12.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:12.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:12.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:12.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:12.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:12.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:12.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:12.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069640 con 0x7f63d8003400 2021-03-09T10:48:12.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:12.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d2f0 con 0x7f63d8009130 2021-03-09T10:48:12.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:12.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001230 con 0x7f63d8005e00 2021-03-09T10:48:12.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:12.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:12.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:12.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:12.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:48:12.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:12.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:12.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:12.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.150+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:12.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:12.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:12.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:12.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:12.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:12.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:12.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:12.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:12.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:48:12.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:48:12.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:12.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:12.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0029870 con 0x7f63d8005e00 2021-03-09T10:48:12.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:12.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:12.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016ed0 con 0x7f63d8005e00 2021-03-09T10:48:12.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:48:12.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.151+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:12.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.152+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:48:12.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.152+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:12.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.152+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19792 cs=0 l=1 rev1=1 rx=0x7f63e007c720 tx=0x7f63fc07b030).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:12.166 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:12.157+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55212 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:12.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc047550 con 0x7f63d8005e00 2021-03-09T10:48:12.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.158+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:12.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.158+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:12.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.158+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:12.168 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:12.168 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:12.168 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:12.168 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:12.169 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:12.169 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:12.169 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:12.169 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:12.170 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:12.170 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:12.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc049700 con 0x7f63d8005e00 2021-03-09T10:48:12.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.158+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:12.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.158+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:48:12.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.159+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049870 con 0x7f63d8005e00 2021-03-09T10:48:12.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.757+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:12.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:12.757+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:13.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:13.757+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:13.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:13.757+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:14.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:14.758+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:14.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:14.758+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:15.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:15.733+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:15.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:15.758+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:15.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:15.758+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:16.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:16.758+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:16.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:16.758+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:17.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:17.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:17.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19792 cs=0 l=1 rev1=1 rx=0x7f63e007c720 tx=0x7f63fc07b030).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:17.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19792 cs=0 l=1 rev1=1 rx=0x7f63e007c720 tx=0x7f63fc07b030).stop 2021-03-09T10:48:17.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:17.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:17.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:17.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=19792 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:17.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:48:17.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:17.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:17.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:17.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:17.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022b20 con 0x7f63d8003400 2021-03-09T10:48:17.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:17.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b600 con 0x7f63d8009130 2021-03-09T10:48:17.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:17.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:48:17.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:17.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:17.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:17.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:17.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.158+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:17.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.159+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:17.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.159+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:48:17.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.159+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:17.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.159+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:17.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.159+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:17.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.159+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:17.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.159+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:17.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:17.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:17.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:17.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:17.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:17.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:48:17.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:48:17.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:17.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:17.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027a50 con 0x7f63d8003840 2021-03-09T10:48:17.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:17.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:17.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:48:17.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:48:17.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.160+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:17.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.161+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:48:17.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.161+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b910 secure :-1 s=READY pgs=19813 cs=0 l=1 rev1=1 rx=0x7f63f000df20 tx=0x7f63f0013b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:17.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.161+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:17.173 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:17.165+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55250 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:17.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0039ed0 con 0x7f63d8003840 2021-03-09T10:48:17.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.166+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:17.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.166+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:17.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.166+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:17.175 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:17.175 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:17.176 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:17.176 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:17.176 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:17.176 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:17.177 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:17.177 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:17.177 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:17.177 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:17.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.168+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0039ed0 con 0x7f63d8003840 2021-03-09T10:48:17.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.168+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:17.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.168+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:48:17.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.169+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0038260 con 0x7f63d8003840 2021-03-09T10:48:17.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.758+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:17.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:17.758+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:18.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:18.758+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:18.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:18.758+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:19.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:19.758+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:19.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:19.758+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:20.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:20.734+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:20.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:20.758+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:20.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:20.758+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:20.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:20.758+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:21.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:21.759+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:21.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:21.759+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:22.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:22.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:22.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b910 secure :-1 s=READY pgs=19813 cs=0 l=1 rev1=1 rx=0x7f63f000df20 tx=0x7f63f0013b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:22.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b910 secure :-1 s=READY pgs=19813 cs=0 l=1 rev1=1 rx=0x7f63f000df20 tx=0x7f63f0013b80).stop 2021-03-09T10:48:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:22.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b910 unknown :-1 s=CLOSED pgs=19813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:48:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:22.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:22.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:22.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:22.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8005e00 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:22.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:22.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014ec0 con 0x7f63d8003400 2021-03-09T10:48:22.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:22.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068da0 con 0x7f63d8009130 2021-03-09T10:48:22.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:22.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8005e00 2021-03-09T10:48:22.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.165+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:22.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.166+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:22.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:22.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.166+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:22.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.166+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:22.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.166+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:22.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.166+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:48:22.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.166+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:22.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.166+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:22.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:22.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:22.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:22.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:22.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:22.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:22.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:22.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:22.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:48:22.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:48:22.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:22.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:22.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f63d8005e00 2021-03-09T10:48:22.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:22.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:22.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027a50 con 0x7f63d8005e00 2021-03-09T10:48:22.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:48:22.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.167+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:22.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.168+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:48:22.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.168+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:22.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.168+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19821 cs=0 l=1 rev1=1 rx=0x7f63f0003540 tx=0x7f63f0055520).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:22.181 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:22.172+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55281 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:48:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.173+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.173+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:22.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.173+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:22.183 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:22.184 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:22.184 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:22.184 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:22.184 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:22.185 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:22.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.174+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8005e00 2021-03-09T10:48:22.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.174+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:22.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.174+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:48:22.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d50 con 0x7f63d8005e00 2021-03-09T10:48:22.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.759+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:22.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:22.759+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:23.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:23.758+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:23.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:23.758+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:24.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:24.759+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:24.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:24.759+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:25.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:25.735+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:25.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:25.759+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:25.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:25.759+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:26.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:26.759+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:26.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:26.759+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:27.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:27.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:27.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19821 cs=0 l=1 rev1=1 rx=0x7f63f0003540 tx=0x7f63f0055520).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:27.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=19821 cs=0 l=1 rev1=1 rx=0x7f63f0003540 tx=0x7f63f0055520).stop 2021-03-09T10:48:27.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:27.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:27.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 msgr2=0x7f63e0020730 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:27.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=CLOSED pgs=19821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:27.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:48:27.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:27.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:27.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:27.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:27.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:27.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:27.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:27.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068830 con 0x7f63d8003400 2021-03-09T10:48:27.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:27.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:48:27.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:27.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003840 2021-03-09T10:48:27.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:27.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:27.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:27.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:27.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.173+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:27.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:27.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:48:27.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:27.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:27.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:27.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:48:27.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:48:27.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:27.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:27.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027a50 con 0x7f63d8003840 2021-03-09T10:48:27.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:27.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:27.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0029870 con 0x7f63d8003840 2021-03-09T10:48:27.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:48:27.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.174+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:27.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.175+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=19835 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63fc04a6d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:27.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.175+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:48:27.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.175+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:27.189 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:27.182+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55329 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:27.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.183+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63d8003840 2021-03-09T10:48:27.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.183+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:27.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.183+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.183+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:27.191 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:27.192 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:27.192 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:27.192 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:27.193 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:27.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.186+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63d8003840 2021-03-09T10:48:27.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.186+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:27.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.186+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:48:27.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.188+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02ddf0 con 0x7f63d8003840 2021-03-09T10:48:27.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.759+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:27.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:27.759+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:28.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:28.760+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:28.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:28.760+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:29.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2021-03-09T10:48:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4049000).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:29.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4049000).stop 2021-03-09T10:48:29.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting, fault. 2021-03-09T10:48:29.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:29.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:29.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:29.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e0020730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f6405d52700 10 monclient: get_auth_request con 0x7f63d8005e00 auth_method 0 2021-03-09T10:48:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:6832/29873,v1:172.21.15.99:6833/29873] conn(0x7f63d8005e00 0x7f63e0020730 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f63f00066c0 tx=0x7f63f0055520).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:29.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.351+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8005e00 2021-03-09T10:48:29.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.760+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:29.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:29.760+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:30.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:30.736+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:30.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:30.760+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:30.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:30.760+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:30.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:30.760+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:31.760+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:31.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:31.760+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:32.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:32.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:32.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=19835 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63fc04a6d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:32.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 secure :-1 s=READY pgs=19835 cs=0 l=1 rev1=1 rx=0x7f63e003b200 tx=0x7f63fc04a6d0).stop 2021-03-09T10:48:32.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:32.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:32.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0008650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:32.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0008650 unknown :-1 s=CLOSED pgs=19835 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:32.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:48:32.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:32.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:32.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:32.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:32.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:32.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:32.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:32.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007bc10 con 0x7f63d8003400 2021-03-09T10:48:32.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:32.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068da0 con 0x7f63d8009130 2021-03-09T10:48:32.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:32.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:48:32.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:32.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:32.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.182+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:32.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.183+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.183+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.183+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.183+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:48:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.183+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:32.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.183+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:32.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.184+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:32.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.184+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:32.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.184+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:32.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:32.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:32.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:32.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:32.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:32.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:48:32.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:48:32.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:32.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:32.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:48:32.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:32.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f63e0018390 2021-03-09T10:48:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:48:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19853 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f007ed20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:32.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:48:32.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.185+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:32.197 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:32.190+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55370 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:32.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63e0018390 2021-03-09T10:48:32.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.191+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:32.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.191+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:32.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.191+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:32.199 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:32.199 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:32.199 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:32.199 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:32.200 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:32.200 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:32.200 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:32.200 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:32.201 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.194+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63e0018390 2021-03-09T10:48:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.194+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.194+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:48:32.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.195+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:48:32.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.760+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:32.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:32.760+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:33.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:33.760+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:33.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:33.760+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:34.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:34.760+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:34.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:34.760+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:35.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:35.737+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:35.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:35.761+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:35.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:35.761+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:36.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:36.761+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:36.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:36.761+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:37.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.190+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:37.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.190+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:37.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.190+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19853 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f007ed20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:37.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.190+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19853 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f007ed20).stop 2021-03-09T10:48:37.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:37.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:37.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:37.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=19853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:37.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:48:37.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:37.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:37.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:37.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:37.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:37.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:37.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:37.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:48:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:48:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:48:37.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:37.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:37.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:37.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:37.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.191+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:37.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.192+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:37.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.192+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:48:37.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.192+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:37.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.192+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:37.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.192+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:37.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.192+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:37.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.192+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:37.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:37.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:37.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:37.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:37.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:37.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:48:37.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:48:37.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:37.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:37.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b7b0 con 0x7f63d8003840 2021-03-09T10:48:37.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:37.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:37.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027a50 con 0x7f63d8003840 2021-03-09T10:48:37.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:48:37.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:37.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:48:37.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:37.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.193+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=19862 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc065790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:37.206 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:37.198+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55408 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.198+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01f420 con 0x7f63d8003840 2021-03-09T10:48:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.198+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.198+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.198+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:37.207 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:37.208 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:37.208 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:37.208 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:37.208 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:37.209 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:37.209 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:37.209 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:37.209 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:37.209 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:37.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc021250 con 0x7f63d8003840 2021-03-09T10:48:37.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.199+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:37.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.199+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:48:37.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.200+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f420 con 0x7f63d8003840 2021-03-09T10:48:37.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.761+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:37.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:37.761+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:38.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:38.761+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:38.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:38.761+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:39.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:39.761+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:39.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:39.761+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:40.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:40.738+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:40.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:40.761+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:40.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:40.761+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:40.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:40.761+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:41.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:41.761+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:41.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:41.761+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:42.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:42.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:42.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=19862 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc065790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:42.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 secure :-1 s=READY pgs=19862 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc065790).stop 2021-03-09T10:48:42.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:42.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:42.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00164d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:42.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00164d0 unknown :-1 s=CLOSED pgs=19862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:42.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:48:42.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:42.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:42.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:42.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:42.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:42.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:42.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:42.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068da0 con 0x7f63d8003400 2021-03-09T10:48:42.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:42.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:48:42.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:42.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f63e0018390 2021-03-09T10:48:42.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:42.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:42.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:42.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:42.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.199+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:42.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.200+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:42.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.200+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:48:42.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.200+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:42.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.200+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.200+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.200+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.200+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:42.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:42.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:42.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:42.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:42.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:48:42.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:48:42.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:42.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:42.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce40 con 0x7f63e0018390 2021-03-09T10:48:42.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:42.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:42.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b7b0 con 0x7f63e0018390 2021-03-09T10:48:42.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:48:42.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.201+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:42.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.202+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:48:42.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.202+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:42.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.202+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19882 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc029f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:42.213 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:42.207+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55449 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:42.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02df10 con 0x7f63e0018390 2021-03-09T10:48:42.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.208+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:42.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.208+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:42.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.208+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:42.215 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:42.215 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:42.216 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:42.216 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:42.216 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:42.216 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:42.216 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:42.217 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:42.217 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:42.217 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:42.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.211+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02df10 con 0x7f63e0018390 2021-03-09T10:48:42.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.211+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:42.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.211+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:48:42.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.213+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065450 con 0x7f63e0018390 2021-03-09T10:48:42.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.761+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:42.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:42.761+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:43.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:43.762+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:43.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:43.762+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:44.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:44.762+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:44.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:44.762+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:45.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:45.739+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:45.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:45.761+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:45.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:45.761+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:46.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:46.762+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:46.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:46.762+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:47.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:48:47.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:47.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19882 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc029f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:47.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19882 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc029f60).stop 2021-03-09T10:48:47.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:47.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:47.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:47.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=19882 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:47.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:48:47.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00680c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:47.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:47.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:47.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:47.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:47.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:47.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:47.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8003400 2021-03-09T10:48:47.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:47.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:48:47.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:47.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068da0 con 0x7f63d8003840 2021-03-09T10:48:47.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.208+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:47.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:47.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:47.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:47.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00680c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:47.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:47.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:48:47.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:47.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:47.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:47.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:47.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.209+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:47.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.210+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:47.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.210+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:47.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.210+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:47.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.210+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:47.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.210+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:47.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.210+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:48:47.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.210+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:48:47.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.210+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:47.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.210+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:47.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.211+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00098c0 con 0x7f63d8003840 2021-03-09T10:48:47.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.211+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:47.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.211+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:47.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.211+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8003840 2021-03-09T10:48:47.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.211+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:48:47.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.211+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:47.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.211+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00680c0 secure :-1 s=READY pgs=19894 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:47.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.211+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:48:47.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.211+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:47.225 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:47.215+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55487 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:47.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005a80 con 0x7f63d8003840 2021-03-09T10:48:47.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.216+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:47.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.216+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:47.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.216+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:47.227 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:47.227 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:47.227 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:47.227 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:47.228 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:47.228 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:47.228 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:47.228 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:47.228 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:47.229 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:47.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.218+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005480 con 0x7f63d8003840 2021-03-09T10:48:47.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.218+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:47.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.219+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:48:47.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.220+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a80 con 0x7f63d8003840 2021-03-09T10:48:47.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.762+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:47.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:47.762+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:48.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:48.762+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:48.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:48.762+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:49.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:49.762+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:49.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:49.762+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:50.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:50.740+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:50.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:50.762+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:50.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:50.762+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:50.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:50.762+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:51.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:51.762+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:51.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:51.762+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:52.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00680c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00680c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00680c0 secure :-1 s=READY pgs=19894 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:52.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00680c0 secure :-1 s=READY pgs=19894 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4009a00).stop 2021-03-09T10:48:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00680c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00680c0 unknown :-1 s=CLOSED pgs=19894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:52.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:48:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:52.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:52.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:52.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:52.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:52.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029b80 con 0x7f63d8003400 2021-03-09T10:48:52.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:52.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d830 con 0x7f63d8009130 2021-03-09T10:48:52.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:52.224 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f63e0018390 2021-03-09T10:48:52.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:52.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:52.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:52.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:52.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:52.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:52.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.217+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:48:52.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.217+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:52.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.217+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:52.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.217+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:52.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.217+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:52.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.217+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:52.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:52.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:52.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:52.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:52.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:52.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:48:52.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:48:52.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:52.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:52.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b0b0 con 0x7f63e0018390 2021-03-09T10:48:52.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:52.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:52.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:48:52.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:48:52.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:52.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:48:52.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:52.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.218+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19908 cs=0 l=1 rev1=1 rx=0x7f63f0029880 tx=0x7f63f004dab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:52.232 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:52.222+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55528 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:52.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63e0018390 2021-03-09T10:48:52.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.223+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:52.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.223+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:52.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.223+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:52.234 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:52.234 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:52.234 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:52.235 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:52.236 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.227+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63e0018390 2021-03-09T10:48:52.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.227+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:52.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.227+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:48:52.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.227+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63e0018390 2021-03-09T10:48:52.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.762+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:52.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:52.762+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:53.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:53.763+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:53.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:53.763+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:54.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:54.763+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:54.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:54.763+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:55.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:55.741+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:48:55.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:55.763+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:55.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:55.763+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:56.763+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:56.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:56.763+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:57.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:48:57.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:48:57.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19908 cs=0 l=1 rev1=1 rx=0x7f63f0029880 tx=0x7f63f004dab0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:48:57.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19908 cs=0 l=1 rev1=1 rx=0x7f63f0029880 tx=0x7f63f004dab0).stop 2021-03-09T10:48:57.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:57.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:48:57.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:48:57.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=19908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:48:57.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:48:57.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:57.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:48:57.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:57.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:48:57.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:48:57.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:48:57.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:57.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e9b0 con 0x7f63d8003400 2021-03-09T10:48:57.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:57.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:48:57.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:48:57.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011d70 con 0x7f63d8003840 2021-03-09T10:48:57.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:48:57.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:48:57.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:57.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:48:57.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:48:57.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:48:57.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.223+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:48:57.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.224+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:48:57.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.224+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:48:57.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:48:57.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:48:57.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:48:57.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:48:57.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:57.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:57.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:48:57.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:48:57.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:48:57.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:48:57.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:48:57.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:57.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027a50 con 0x7f63d8003840 2021-03-09T10:48:57.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:48:57.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:48:57.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b0b0 con 0x7f63d8003840 2021-03-09T10:48:57.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:48:57.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.225+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:57.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.226+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:48:57.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.226+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069e80 secure :-1 s=READY pgs=19922 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63f0038750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:48:57.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.226+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:48:57.238 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:48:57.231+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55566 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:48:57.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:48:57.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.232+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:48:57.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.232+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:48:57.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.232+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:48:57.240 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:48:57.240 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:48:57.240 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:57.240 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:48:57.241 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:48:57.241 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:48:57.241 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:48:57.241 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:48:57.241 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:48:57.242 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:48:57.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:48:57.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.237+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:48:57.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.237+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:48:57.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e4a0 con 0x7f63d8003840 2021-03-09T10:48:57.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.763+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:57.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:57.763+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:58.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:58.763+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:58.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:58.763+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:48:59.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:59.764+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:48:59.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:48:59.764+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:00.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:00.742+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:00.764+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:00.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:00.764+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:00.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:00.764+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:01.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:01.764+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:01.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:01.764+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:02.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:02.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:02.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069e80 secure :-1 s=READY pgs=19922 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63f0038750).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:02.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069e80 secure :-1 s=READY pgs=19922 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63f0038750).stop 2021-03-09T10:49:02.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:02.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:02.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0069e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:02.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0069e80 unknown :-1 s=CLOSED pgs=19922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:02.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:49:02.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:02.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:02.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.231+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:02.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:02.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:02.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:02.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:02.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015ab0 con 0x7f63d8003400 2021-03-09T10:49:02.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:02.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d830 con 0x7f63d8009130 2021-03-09T10:49:02.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:02.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63e0018390 2021-03-09T10:49:02.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:02.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:02.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:02.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:02.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:02.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:02.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:49:02.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:02.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:02.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.232+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:02.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:02.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:02.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:02.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:02.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:02.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:02.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:02.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:49:02.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:49:02.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:02.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:02.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005ef50 con 0x7f63e0018390 2021-03-09T10:49:02.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:02.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:02.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce40 con 0x7f63e0018390 2021-03-09T10:49:02.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:49:02.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:02.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:49:02.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19932 cs=0 l=1 rev1=1 rx=0x7f63fc065ec0 tx=0x7f63fc04a7b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:02.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.233+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:02.247 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:02.239+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55622 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:02.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.240+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc008070 con 0x7f63e0018390 2021-03-09T10:49:02.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.240+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:02.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.240+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:02.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.240+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:02.249 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:02.249 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:02.249 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:02.249 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:02.250 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:02.250 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:02.250 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:02.250 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:02.251 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:02.251 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:02.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc008070 con 0x7f63e0018390 2021-03-09T10:49:02.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.244+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:02.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.244+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:49:02.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.245+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc03fd20 con 0x7f63e0018390 2021-03-09T10:49:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.764+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:02.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:02.764+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:03.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:03.763+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:03.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:03.763+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:04.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:04.764+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:04.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:04.764+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:05.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:05.743+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:05.764+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:05.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:05.764+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:06.764+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:06.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:06.764+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:07.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:07.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:07.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19932 cs=0 l=1 rev1=1 rx=0x7f63fc065ec0 tx=0x7f63fc04a7b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:07.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19932 cs=0 l=1 rev1=1 rx=0x7f63fc065ec0 tx=0x7f63fc04a7b0).stop 2021-03-09T10:49:07.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:07.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:07.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:07.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=19932 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:07.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:49:07.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:07.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:07.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:07.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:07.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:07.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:07.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:07.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068da0 con 0x7f63d8003400 2021-03-09T10:49:07.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:07.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d090 con 0x7f63d8009130 2021-03-09T10:49:07.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:07.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:49:07.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:07.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:07.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:07.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:07.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.240+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:07.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.241+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:07.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.241+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:49:07.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.241+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:07.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.241+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:07.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.242+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:07.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.242+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:07.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.242+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:07.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.242+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:07.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:07.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:07.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:07.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:07.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:49:07.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:49:07.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:07.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:07.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0068460 con 0x7f63d8003840 2021-03-09T10:49:07.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:07.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:07.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027a50 con 0x7f63d8003840 2021-03-09T10:49:07.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:49:07.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:07.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.243+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067f70 secure :-1 s=READY pgs=19952 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0038260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:07.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.244+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:49:07.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.244+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:07.254 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:07.248+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55720 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:07.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0000bc0 con 0x7f63d8003840 2021-03-09T10:49:07.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.249+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:07.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.249+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:07.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.249+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:07.256 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:07.257 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:07.257 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:07.257 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:07.257 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:07.257 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:07.258 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:07.258 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:07.258 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:07.258 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:07.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.250+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:49:07.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.250+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:07.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.250+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:49:07.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.251+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:49:07.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.764+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:07.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:07.764+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:08.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:08.765+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:08.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:08.765+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:09.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:09.765+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:09.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:09.765+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:10.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:10.744+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:10.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:10.764+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:10.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:10.764+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:10.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:10.764+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:11.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:11.765+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:11.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:11.765+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:12.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:12.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:12.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067f70 secure :-1 s=READY pgs=19952 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0038260).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:12.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067f70 secure :-1 s=READY pgs=19952 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0038260).stop 2021-03-09T10:49:12.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:12.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:12.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:12.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067f70 unknown :-1 s=CLOSED pgs=19952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:12.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:49:12.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:12.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:12.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:12.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:12.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023fa0 con 0x7f63d8003400 2021-03-09T10:49:12.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:12.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015ab0 con 0x7f63d8009130 2021-03-09T10:49:12.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:12.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014ce0 con 0x7f63e0018390 2021-03-09T10:49:12.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:12.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:12.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.249+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:12.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.250+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:12.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.250+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:12.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.250+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:12.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.250+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:49:12.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.250+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:12.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.250+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:12.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:12.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:12.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:12.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:12.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:12.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:12.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:12.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:12.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:49:12.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:49:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0027a50 con 0x7f63e0018390 2021-03-09T10:49:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:12.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:12.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005ef50 con 0x7f63e0018390 2021-03-09T10:49:12.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:49:12.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.251+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:12.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.252+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19963 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc012fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:12.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.252+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:49:12.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.252+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:12.266 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:12.256+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55821 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:12.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.258+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63e0018390 2021-03-09T10:49:12.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.258+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.258+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.258+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:12.267 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:12.268 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:12.269 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:12.269 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:12.269 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:12.269 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:12.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.260+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63e0018390 2021-03-09T10:49:12.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.260+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:12.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.260+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:49:12.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.262+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:49:12.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.765+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:12.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:12.765+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:13.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:13.765+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:13.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:13.765+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:14.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:14.766+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:14.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:14.766+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:15.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:15.745+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:15.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:15.766+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:15.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:15.766+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:16.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:16.765+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:16.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:16.765+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:17.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19963 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc012fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19963 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc012fb0).stop 2021-03-09T10:49:17.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=19963 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:17.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:49:17.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:17.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:17.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:17.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:17.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:17.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:17.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:17.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017150 con 0x7f63d8003400 2021-03-09T10:49:17.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:17.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e0f0 con 0x7f63d8009130 2021-03-09T10:49:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068da0 con 0x7f63d8003840 2021-03-09T10:49:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:17.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:17.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:17.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.257+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:17.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.258+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:49:17.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.258+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:17.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.258+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:17.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.258+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:17.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.258+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:17.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.258+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:17.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.258+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:17.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:17.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:17.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:17.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:17.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:49:17.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:49:17.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:17.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:17.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001c9b0 con 0x7f63d8003840 2021-03-09T10:49:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0027a50 con 0x7f63d8003840 2021-03-09T10:49:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:49:17.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:17.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:49:17.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:17.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.259+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021d30 secure :-1 s=READY pgs=19975 cs=0 l=1 rev1=1 rx=0x7f63e002bcf0 tx=0x7f63fc06fd50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:17.273 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:17.264+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 55919 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:17.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.266+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013470 con 0x7f63d8003840 2021-03-09T10:49:17.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.266+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:17.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.266+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:17.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.266+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:17.274 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:17.275 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:17.276 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:17.276 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:17.276 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:17.276 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:17.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f63d8003840 2021-03-09T10:49:17.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.269+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:17.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.269+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:49:17.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.270+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013d70 con 0x7f63d8003840 2021-03-09T10:49:17.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.766+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:17.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:17.766+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:18.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:18.765+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:18.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:18.765+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:19.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:19.766+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:19.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:19.766+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:20.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:20.746+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:20.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:20.766+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:20.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:20.766+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:20.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:20.766+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:21.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:21.767+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:21.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:21.767+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:22.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:22.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021d30 secure :-1 s=READY pgs=19975 cs=0 l=1 rev1=1 rx=0x7f63e002bcf0 tx=0x7f63fc06fd50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021d30 secure :-1 s=READY pgs=19975 cs=0 l=1 rev1=1 rx=0x7f63e002bcf0 tx=0x7f63fc06fd50).stop 2021-03-09T10:49:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:22.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0021d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:22.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0021d30 unknown :-1 s=CLOSED pgs=19975 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:22.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:49:22.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:22.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:22.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:22.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:22.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:22.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:22.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:22.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016f50 con 0x7f63d8003400 2021-03-09T10:49:22.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:22.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00160e0 con 0x7f63d8009130 2021-03-09T10:49:22.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:22.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068fa0 con 0x7f63e0018390 2021-03-09T10:49:22.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:22.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:22.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:22.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:22.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:22.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:22.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:49:22.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:22.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:22.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:22.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:22.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.265+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:22.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:22.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:22.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:49:22.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:49:22.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:22.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:22.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066420 con 0x7f63e0018390 2021-03-09T10:49:22.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:22.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:22.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0068460 con 0x7f63e0018390 2021-03-09T10:49:22.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:49:22.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.266+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:22.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.268+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19993 cs=0 l=1 rev1=1 rx=0x7f63e0043e20 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:22.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.268+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:49:22.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.268+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:22.281 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:22.272+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56009 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:22.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63e0018390 2021-03-09T10:49:22.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.273+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:22.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.273+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:22.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.273+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:22.283 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:22.283 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:22.283 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:22.283 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:22.284 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:22.284 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:22.284 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:22.284 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:22.285 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:22.285 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:22.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63e0018390 2021-03-09T10:49:22.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.273+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:22.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.273+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:49:22.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d30 con 0x7f63e0018390 2021-03-09T10:49:22.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.767+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:22.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:22.767+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:23.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:23.766+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:23.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:23.766+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:24.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:24.767+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:24.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:24.767+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:25.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:25.747+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:25.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:25.767+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:25.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:25.767+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:26.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:26.767+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:26.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:26.767+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:27.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:27.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:27.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19993 cs=0 l=1 rev1=1 rx=0x7f63e0043e20 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:27.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=19993 cs=0 l=1 rev1=1 rx=0x7f63e0043e20 tx=0x7f63f004de00).stop 2021-03-09T10:49:27.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:27.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:27.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:27.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=19993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:27.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:49:27.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:27.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:27.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:27.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:27.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:27.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:27.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:27.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:49:27.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:27.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068da0 con 0x7f63d8009130 2021-03-09T10:49:27.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:27.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:49:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:27.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.273+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:27.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:27.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:49:27.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:27.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:27.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:27.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:27.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:27.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0024560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:27.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:49:27.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:49:27.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:27.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:27.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e2a0 con 0x7f63d8003840 2021-03-09T10:49:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001c9b0 con 0x7f63d8003840 2021-03-09T10:49:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:49:27.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.274+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:27.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.276+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 secure :-1 s=READY pgs=20005 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc02df80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:27.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.276+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:49:27.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.276+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:27.288 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:27.281+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56118 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:27.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.281+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06fe90 con 0x7f63d8003840 2021-03-09T10:49:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.281+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.281+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.281+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:27.289 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:27.290 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:27.291 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:27.291 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:27.291 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:27.291 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:27.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.281+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc013d70 con 0x7f63d8003840 2021-03-09T10:49:27.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.281+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:27.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.281+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:49:27.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc013290 con 0x7f63d8003840 2021-03-09T10:49:27.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.767+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:27.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:27.767+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:28.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:28.768+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:28.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:28.768+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:29.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:29.768+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:29.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:29.768+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:30.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:30.748+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:30.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:30.768+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:30.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:30.768+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:30.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:30.768+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:31.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:31.767+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:31.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:31.767+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:32.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.281+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:32.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.281+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:32.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.281+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 secure :-1 s=READY pgs=20005 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc02df80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:32.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.281+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 secure :-1 s=READY pgs=20005 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc02df80).stop 2021-03-09T10:49:32.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.281+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:32.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:32.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:32.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 unknown :-1 s=CLOSED pgs=20005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:32.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:49:32.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:32.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:32.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:32.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:32.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:32.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:32.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:32.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63d8003400 2021-03-09T10:49:32.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:32.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00160e0 con 0x7f63d8009130 2021-03-09T10:49:32.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:32.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065c10 con 0x7f63e0018390 2021-03-09T10:49:32.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:32.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:32.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:32.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:32.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:32.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:32.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:49:32.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:32.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.282+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:32.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:32.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:32.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:32.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:32.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:32.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:32.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:32.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:32.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:49:32.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:49:32.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:32.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:32.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006de70 con 0x7f63e0018390 2021-03-09T10:49:32.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:32.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:32.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066420 con 0x7f63e0018390 2021-03-09T10:49:32.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:49:32.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.283+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:32.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024560 secure :-1 s=READY pgs=20016 cs=0 l=1 rev1=1 rx=0x7f63e007afc0 tx=0x7f63f0057f60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:32.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.284+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:49:32.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.284+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:32.297 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:32.289+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56221 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:32.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.291+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63e0018390 2021-03-09T10:49:32.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.291+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:32.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.291+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:32.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.291+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:32.299 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:32.299 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:32.299 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:32.299 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:32.300 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:32.300 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:32.300 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:32.300 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:32.300 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:32.301 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:32.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.291+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0011b10 con 0x7f63e0018390 2021-03-09T10:49:32.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.291+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:32.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.291+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:49:32.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.292+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63e0018390 2021-03-09T10:49:32.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.768+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:32.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:32.768+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:33.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:33.767+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:33.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:33.767+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:34.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:34.768+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:34.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:34.768+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:35.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:35.749+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:35.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:35.768+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:35.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:35.768+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:36.769+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:36.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:36.769+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:37.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0024560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:37.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0024560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:37.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024560 secure :-1 s=READY pgs=20016 cs=0 l=1 rev1=1 rx=0x7f63e007afc0 tx=0x7f63f0057f60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024560 secure :-1 s=READY pgs=20016 cs=0 l=1 rev1=1 rx=0x7f63e007afc0 tx=0x7f63f0057f60).stop 2021-03-09T10:49:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:37.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0024560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:37.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0024560 unknown :-1 s=CLOSED pgs=20016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:37.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:49:37.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:37.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:37.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:37.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:37.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:37.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:37.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:49:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016710 con 0x7f63d8009130 2021-03-09T10:49:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:37.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63d8003840 2021-03-09T10:49:37.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:37.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:37.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:37.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:37.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.290+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:37.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.291+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:37.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.291+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:49:37.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.291+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.291+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:37.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:37.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:37.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:37.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:37.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:37.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:49:37.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:49:37.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:37.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:37.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007c920 con 0x7f63d8003840 2021-03-09T10:49:37.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:37.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:37.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006de70 con 0x7f63d8003840 2021-03-09T10:49:37.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:49:37.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.292+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:37.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.293+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:49:37.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.293+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 secure :-1 s=READY pgs=20033 cs=0 l=1 rev1=1 rx=0x7f63f007f650 tx=0x7f63f0031f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:37.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.293+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:37.305 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:37.299+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56319 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:37.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.299+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:49:37.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.299+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:37.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.299+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:37.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.299+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:37.307 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:37.307 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:37.307 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:37.308 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:37.308 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:37.308 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:37.308 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:37.309 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:37.309 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:37.309 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:37.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.301+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007f2b0 con 0x7f63d8003840 2021-03-09T10:49:37.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.301+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:37.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.301+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:49:37.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.302+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005f2e0 con 0x7f63d8003840 2021-03-09T10:49:37.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.769+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:37.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:37.769+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:38.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:38.769+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:38.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:38.769+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:39.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:39.769+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:39.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:39.769+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:40.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:40.750+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:40.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:40.769+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:40.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:40.769+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:40.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:40.769+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:41.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:41.768+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:41.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:41.768+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:42.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 secure :-1 s=READY pgs=20033 cs=0 l=1 rev1=1 rx=0x7f63f007f650 tx=0x7f63f0031f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 secure :-1 s=READY pgs=20033 cs=0 l=1 rev1=1 rx=0x7f63f007f650 tx=0x7f63f0031f50).stop 2021-03-09T10:49:42.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067ed0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067ed0 unknown :-1 s=CLOSED pgs=20033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:42.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:49:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:42.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:42.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:42.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:42.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:49:42.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:42.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067bb0 con 0x7f63d8009130 2021-03-09T10:49:42.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:42.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63e0018390 2021-03-09T10:49:42.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:42.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:42.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:49:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:42.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:42.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.300+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:42.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:42.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:42.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:42.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:42.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:42.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:49:42.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:49:42.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:42.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:42.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001c9b0 con 0x7f63e0018390 2021-03-09T10:49:42.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:42.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:42.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e2a0 con 0x7f63e0018390 2021-03-09T10:49:42.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:49:42.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:42.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:49:42.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:42.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.301+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 secure :-1 s=READY pgs=20042 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc008070).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:42.316 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:42.307+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56420 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:42.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.307+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc01e2e0 con 0x7f63e0018390 2021-03-09T10:49:42.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.308+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:42.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.308+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:42.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.308+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:42.318 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:42.318 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:42.318 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:42.319 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:42.319 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:42.319 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:42.319 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:42.319 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:42.320 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:42.320 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:42.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01e440 con 0x7f63e0018390 2021-03-09T10:49:42.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.308+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:42.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.308+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:49:42.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.309+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc012240 con 0x7f63e0018390 2021-03-09T10:49:42.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.769+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:42.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:42.769+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:43.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:43.770+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:43.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:43.770+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:44.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:44.770+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:44.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:44.770+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:45.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:45.751+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:45.770+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:45.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:45.770+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:46.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:46.770+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:46.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:46.770+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:47.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:49:47.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:47.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 secure :-1 s=READY pgs=20042 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc008070).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:47.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 secure :-1 s=READY pgs=20042 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc008070).stop 2021-03-09T10:49:47.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:47.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:47.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:47.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 unknown :-1 s=CLOSED pgs=20042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:47.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:49:47.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:47.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:47.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:47.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:47.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:47.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:47.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:47.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0067bb0 con 0x7f63d8003400 2021-03-09T10:49:47.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:47.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001270 con 0x7f63d8009130 2021-03-09T10:49:47.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:47.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63d8003840 2021-03-09T10:49:47.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:47.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:47.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:47.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:47.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.308+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:47.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.309+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:47.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.309+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:49:47.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.309+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:47.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.309+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:47.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.309+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:47.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.309+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:47.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.309+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:47.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:47.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:47.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:47.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:47.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:47.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:49:47.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:49:47.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:47.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:47.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:49:47.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:47.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:47.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007c920 con 0x7f63d8003840 2021-03-09T10:49:47.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:49:47.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.310+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:47.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.311+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067e80 secure :-1 s=READY pgs=20062 cs=0 l=1 rev1=1 rx=0x7f63f0036510 tx=0x7f63f0000c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:47.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.311+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:49:47.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.311+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:47.324 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:47.316+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56518 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:47.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.316+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8003840 2021-03-09T10:49:47.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.316+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:47.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.316+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:47.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.316+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:47.326 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:47.326 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:47.326 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:47.327 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:47.327 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:47.327 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:47.327 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:47.328 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:47.328 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:47.328 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:47.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.320+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0036170 con 0x7f63d8003840 2021-03-09T10:49:47.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.320+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:47.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.320+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:49:47.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.321+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ed20 con 0x7f63d8003840 2021-03-09T10:49:47.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.770+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:47.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:47.770+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:48.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:48.770+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:48.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:48.770+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:49.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:49.770+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:49.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:49.770+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:50.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:50.752+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:50.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:50.771+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:50.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:50.771+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:50.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:50.771+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:51.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:51.771+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:51.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:51.771+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:52.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:52.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:52.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067e80 secure :-1 s=READY pgs=20062 cs=0 l=1 rev1=1 rx=0x7f63f0036510 tx=0x7f63f0000c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:52.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067e80 secure :-1 s=READY pgs=20062 cs=0 l=1 rev1=1 rx=0x7f63f0036510 tx=0x7f63f0000c30).stop 2021-03-09T10:49:52.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:52.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:52.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:52.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067e80 unknown :-1 s=CLOSED pgs=20062 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:52.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:49:52.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:52.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:52.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:52.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:52.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:52.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:52.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:52.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:49:52.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:52.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63d8009130 2021-03-09T10:49:52.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:52.324 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027e50 con 0x7f63e0018390 2021-03-09T10:49:52.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:52.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:52.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:52.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:52.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.317+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:52.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:52.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:49:52.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:52.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:52.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:52.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:52.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:52.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:49:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:49:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:52.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067bb0 con 0x7f63e0018390 2021-03-09T10:49:52.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:52.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:52.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63e0018390 2021-03-09T10:49:52.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:49:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.319+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:52.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.320+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:49:52.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.320+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20073 cs=0 l=1 rev1=1 rx=0x7f63f004db60 tx=0x7f63f00393a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:52.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.320+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:52.340 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:52.326+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56619 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:52.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.326+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f002f3e0 con 0x7f63e0018390 2021-03-09T10:49:52.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.326+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:52.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.326+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:52.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.326+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:52.343 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:52.343 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:52.343 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:52.344 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:52.344 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:52.345 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:52.345 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:52.345 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:52.346 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:52.346 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:52.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.328+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f002f3e0 con 0x7f63e0018390 2021-03-09T10:49:52.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.328+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:52.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.328+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:49:52.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.329+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f00155e0 con 0x7f63e0018390 2021-03-09T10:49:52.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.771+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:52.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:52.771+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:53.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:53.771+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:53.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:53.771+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:54.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:54.771+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:54.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:54.771+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:55.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:55.753+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:49:55.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:55.771+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:55.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:55.771+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:56.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:56.771+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:56.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:56.771+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:57.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.326+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:49:57.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.326+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:49:57.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.326+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20073 cs=0 l=1 rev1=1 rx=0x7f63f004db60 tx=0x7f63f00393a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:49:57.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.326+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20073 cs=0 l=1 rev1=1 rx=0x7f63f004db60 tx=0x7f63f00393a0).stop 2021-03-09T10:49:57.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:57.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:49:57.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:49:57.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20073 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:49:57.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:49:57.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:57.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:49:57.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:57.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:49:57.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:49:57.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:49:57.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:57.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:49:57.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:57.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:49:57.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:49:57.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017780 con 0x7f63d8003840 2021-03-09T10:49:57.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:49:57.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:49:57.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:57.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:49:57.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.327+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:49:57.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.328+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:49:57.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.328+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:49:57.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.328+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:49:57.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.328+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:49:57.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:49:57.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:49:57.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.328+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:49:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:49:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:57.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:57.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:49:57.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:49:57.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:49:57.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:49:57.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:49:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010d70 con 0x7f63d8003840 2021-03-09T10:49:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:49:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:49:57.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067bb0 con 0x7f63d8003840 2021-03-09T10:49:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:49:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.329+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.330+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068560 secure :-1 s=READY pgs=20085 cs=0 l=1 rev1=1 rx=0x7f63f00298d0 tx=0x7f63f0039480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:49:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.330+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:49:57.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.330+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:49:57.341 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:49:57.334+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56717 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:49:57.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:49:57.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.335+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:49:57.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.335+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:49:57.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.335+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:49:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:49:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:49:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:49:57.343 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:49:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:49:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:49:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:49:57.344 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:49:57.344 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:49:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:49:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.335+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:49:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.335+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:49:57.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.337+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63d8003840 2021-03-09T10:49:57.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.772+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:57.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:57.772+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:58.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:58.772+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:58.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:58.772+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:49:59.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:59.772+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:49:59.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:49:59.772+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:00.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:00.754+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:00.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:00.772+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:00.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:00.772+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:00.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:00.772+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:01.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:01.772+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:01.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:01.772+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:02.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:02.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:02.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068560 secure :-1 s=READY pgs=20085 cs=0 l=1 rev1=1 rx=0x7f63f00298d0 tx=0x7f63f0039480).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:02.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068560 secure :-1 s=READY pgs=20085 cs=0 l=1 rev1=1 rx=0x7f63f00298d0 tx=0x7f63f0039480).stop 2021-03-09T10:50:02.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:02.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:02.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:02.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068560 unknown :-1 s=CLOSED pgs=20085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:02.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:50:02.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:02.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:02.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:02.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:02.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:02.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:02.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:02.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:50:02.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:02.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021700 con 0x7f63d8009130 2021-03-09T10:50:02.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:02.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00228b0 con 0x7f63e0018390 2021-03-09T10:50:02.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:02.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:02.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:02.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:02.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:02.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.335+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:02.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:50:02.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:02.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:02.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:02.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:02.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:02.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:02.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:02.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:02.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:02.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:02.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:50:02.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:50:02.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:02.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:02.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e690 con 0x7f63e0018390 2021-03-09T10:50:02.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:02.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:02.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0010d70 con 0x7f63e0018390 2021-03-09T10:50:02.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:50:02.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:02.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.336+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20096 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0090350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:02.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.337+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:50:02.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.337+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:02.350 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:02.342+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56819 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:02.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63e0018390 2021-03-09T10:50:02.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.343+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:02.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.343+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:02.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.343+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:02.352 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:02.352 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:02.352 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:02.352 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:02.353 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:02.353 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:02.353 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:02.353 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:02.353 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:02.354 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:02.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63e0018390 2021-03-09T10:50:02.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.346+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:02.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.346+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:50:02.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.347+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0023310 con 0x7f63e0018390 2021-03-09T10:50:02.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.772+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:02.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:02.772+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:03.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:03.772+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:03.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:03.772+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:04.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:04.772+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:04.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:04.772+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:05.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:05.755+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:05.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:05.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:05.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:05.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:06.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:06.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:06.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:06.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:07.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.342+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:07.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.342+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:07.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.342+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20096 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0090350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:07.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.342+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20096 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0090350).stop 2021-03-09T10:50:07.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.342+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:07.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.342+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:07.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:07.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:07.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:50:07.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:07.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:07.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:07.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:07.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:07.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:07.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:07.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63d8003400 2021-03-09T10:50:07.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:07.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce70 con 0x7f63d8009130 2021-03-09T10:50:07.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:07.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b9c0 con 0x7f63d8003840 2021-03-09T10:50:07.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:07.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:07.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:07.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:07.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:07.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:07.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:50:07.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:07.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.343+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:07.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:07.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:07.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:07.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:07.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:07.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:07.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:07.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:07.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:50:07.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:50:07.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:07.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:07.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007a3a0 con 0x7f63d8003840 2021-03-09T10:50:07.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:07.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:07.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e690 con 0x7f63d8003840 2021-03-09T10:50:07.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:50:07.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.344+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:07.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.345+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 secure :-1 s=READY pgs=20111 cs=0 l=1 rev1=1 rx=0x7f63e00227c0 tx=0x7f63f0029370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:07.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.345+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:50:07.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.345+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:07.357 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:07.351+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 56917 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:07.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:50:07.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.352+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:07.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.352+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:07.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.352+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:07.358 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:07.358 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:07.359 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:07.359 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:07.359 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:07.359 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:07.359 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:07.360 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:07.360 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:07.360 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:07.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.352+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0090740 con 0x7f63d8003840 2021-03-09T10:50:07.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.352+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:07.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.352+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:50:07.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.353+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0011da0 con 0x7f63d8003840 2021-03-09T10:50:07.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:07.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:07.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:08.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:08.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:08.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:08.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:09.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:09.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:09.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:09.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:10.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:10.756+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:10.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:10.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:10.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:10.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:10.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:10.773+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:11.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:11.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:11.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:11.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:12.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:12.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:12.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 secure :-1 s=READY pgs=20111 cs=0 l=1 rev1=1 rx=0x7f63e00227c0 tx=0x7f63f0029370).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:12.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 secure :-1 s=READY pgs=20111 cs=0 l=1 rev1=1 rx=0x7f63e00227c0 tx=0x7f63f0029370).stop 2021-03-09T10:50:12.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:12.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:12.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:12.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 unknown :-1 s=CLOSED pgs=20111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:12.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:50:12.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:12.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:12.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:12.355 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:12.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:12.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:12.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:12.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T10:50:12.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:12.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021700 con 0x7f63d8009130 2021-03-09T10:50:12.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:12.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:50:12.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:12.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:12.358 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.351+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:12.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:12.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:12.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:12.359 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:50:12.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:12.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:12.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:12.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:12.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:12.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.352+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:12.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:12.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:12.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:12.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:12.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:50:12.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:50:12.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:12.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:12.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00266a0 con 0x7f63e0018390 2021-03-09T10:50:12.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:12.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:12.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00098c0 con 0x7f63e0018390 2021-03-09T10:50:12.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:50:12.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:12.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:50:12.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:12.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.353+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20124 cs=0 l=1 rev1=1 rx=0x7f63f400dc20 tx=0x7f63f400dad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:12.365 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:12.358+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57018 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:12.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.359+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T10:50:12.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.359+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:12.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.359+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:12.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.359+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:12.367 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:12.368 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:12.369 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.362+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T10:50:12.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.362+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.362+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:50:12.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.363+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T10:50:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:12.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:13.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:13.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:13.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:13.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:14.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:14.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:14.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:14.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:15.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:15.757+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:15.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:15.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:15.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:15.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:16.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:16.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:16.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:16.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:17.360 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:50:17.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:17.361 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20124 cs=0 l=1 rev1=1 rx=0x7f63f400dc20 tx=0x7f63f400dad0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:17.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20124 cs=0 l=1 rev1=1 rx=0x7f63f400dc20 tx=0x7f63f400dad0).stop 2021-03-09T10:50:17.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:17.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:17.362 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:17.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:17.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:50:17.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:17.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:17.363 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:17.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:17.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:17.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:17.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:17.364 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63d8003400 2021-03-09T10:50:17.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:17.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c1f0 con 0x7f63d8009130 2021-03-09T10:50:17.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:17.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003840 2021-03-09T10:50:17.365 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:17.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:17.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:17.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:17.366 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.358+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:17.367 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.359+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:17.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.359+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:50:17.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.359+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:17.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.359+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:17.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.360+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:17.368 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.360+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:17.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.360+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:17.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:17.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:17.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:17.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:17.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:17.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:50:17.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:50:17.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:17.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:17.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:50:17.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00266a0 con 0x7f63d8003840 2021-03-09T10:50:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:50:17.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.361+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=20144 cs=0 l=1 rev1=1 rx=0x7f63f4072c80 tx=0x7f63f40226c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.362+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:50:17.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.362+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:17.373 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:17.365+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57116 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:17.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.368+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400ee80 con 0x7f63d8003840 2021-03-09T10:50:17.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.368+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.368+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.368+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:17.375 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:17.376 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:17.376 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:17.376 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:17.376 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:17.377 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:17.377 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:17.377 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:17.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.369+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400ee80 con 0x7f63d8003840 2021-03-09T10:50:17.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.369+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:17.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.370+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:50:17.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.370+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63d8003840 2021-03-09T10:50:17.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:17.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:17.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:18.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:18.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:18.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:18.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:19.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:19.773+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:19.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:19.773+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:20.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:20.757+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:20.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:20.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:20.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:20.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:20.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:20.774+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:21.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:21.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:21.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:21.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:22.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:50:22.369 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:22.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=20144 cs=0 l=1 rev1=1 rx=0x7f63f4072c80 tx=0x7f63f40226c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:22.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=20144 cs=0 l=1 rev1=1 rx=0x7f63f4072c80 tx=0x7f63f40226c0).stop 2021-03-09T10:50:22.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:22.370 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:22.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:22.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=CLOSED pgs=20144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:22.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:50:22.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:22.371 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:22.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:22.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:22.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:22.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:22.372 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:22.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e2f0 con 0x7f63d8003400 2021-03-09T10:50:22.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:22.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00161a0 con 0x7f63d8009130 2021-03-09T10:50:22.373 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:22.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:50:22.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:22.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:22.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.366+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:22.374 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.367+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:22.375 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.367+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:22.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:22.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:50:22.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:22.376 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:22.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:22.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:50:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:50:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002e320 con 0x7f63e0018390 2021-03-09T10:50:22.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:22.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:22.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001c9b0 con 0x7f63e0018390 2021-03-09T10:50:22.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:50:22.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.368+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:22.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.369+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20155 cs=0 l=1 rev1=1 rx=0x7f63e0030040 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:22.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.369+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:50:22.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.369+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:22.381 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:22.374+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57216 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:22.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058c30 con 0x7f63e0018390 2021-03-09T10:50:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.375+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.375+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.375+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:22.383 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:22.384 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:22.385 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.376+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:50:22.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.376+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.377+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:50:22.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.377+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc01f340 con 0x7f63e0018390 2021-03-09T10:50:22.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:22.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:22.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:23.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:23.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:23.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:23.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:24.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:24.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:24.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:24.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:25.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:25.758+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:25.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:25.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:25.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:25.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:26.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:26.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:26.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:26.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:27.377 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:27.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:27.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20155 cs=0 l=1 rev1=1 rx=0x7f63e0030040 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:27.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20155 cs=0 l=1 rev1=1 rx=0x7f63e0030040 tx=0x7f63fc04a740).stop 2021-03-09T10:50:27.378 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:50:27.379 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:27.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:27.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:27.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:27.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:27.380 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021700 con 0x7f63d8003400 2021-03-09T10:50:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63d8009130 2021-03-09T10:50:27.381 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:27.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:50:27.382 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:27.383 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:27.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:27.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:27.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.375+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:27.384 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.376+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:27.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.376+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:50:27.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.376+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:27.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.376+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:27.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.376+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:27.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.376+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:27.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.376+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:27.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:27.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:27.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:50:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:50:27.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:27.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:27.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00104b0 con 0x7f63d8003840 2021-03-09T10:50:27.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:27.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:27.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002e320 con 0x7f63d8003840 2021-03-09T10:50:27.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:50:27.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.377+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:27.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.378+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b460 secure :-1 s=READY pgs=20170 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc078ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:27.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.378+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:50:27.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.378+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:27.390 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:27.382+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57315 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:27.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065450 con 0x7f63d8003840 2021-03-09T10:50:27.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.383+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:27.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.383+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:27.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.383+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:27.392 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:27.392 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:27.392 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:27.392 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:27.393 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:27.393 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:27.393 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:27.393 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:27.393 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:27.394 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:27.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.387+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63d8003840 2021-03-09T10:50:27.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.387+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:27.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.388+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:50:27.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.389+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065450 con 0x7f63d8003840 2021-03-09T10:50:27.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.774+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:27.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:27.774+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:28.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:28.775+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:28.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:28.775+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:29.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:29.775+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:29.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:29.775+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:30.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:30.759+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:30.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:30.775+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:30.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:30.775+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:30.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:30.775+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:31.775+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:31.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:31.775+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:32.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:32.385 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:32.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b460 secure :-1 s=READY pgs=20170 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc078ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:32.386 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b460 secure :-1 s=READY pgs=20170 cs=0 l=1 rev1=1 rx=0x7f63fc06f070 tx=0x7f63fc078ec0).stop 2021-03-09T10:50:32.387 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:32.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:32.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001b460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:32.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001b460 unknown :-1 s=CLOSED pgs=20170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:32.388 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:50:32.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:32.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:32.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:32.389 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:32.390 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016910 con 0x7f63d8003400 2021-03-09T10:50:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:50:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:32.391 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63e0018390 2021-03-09T10:50:32.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:32.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:32.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:32.392 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.383+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.384+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.384+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:50:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.384+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:32.393 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.384+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.384+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.384+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.384+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:32.394 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:32.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:32.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:32.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:32.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:50:32.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:50:32.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:32.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:32.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0011d50 con 0x7f63e0018390 2021-03-09T10:50:32.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:32.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:32.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00104b0 con 0x7f63e0018390 2021-03-09T10:50:32.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:50:32.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.385+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:32.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.386+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20180 cs=0 l=1 rev1=1 rx=0x7f63e0028390 tx=0x7f63fc066f10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:32.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.386+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:50:32.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.386+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:32.399 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:32.391+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57416 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:32.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.391+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc033820 con 0x7f63e0018390 2021-03-09T10:50:32.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.391+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:32.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.391+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:32.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.391+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:32.400 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:32.400 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:32.401 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:32.401 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:32.401 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:32.401 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:32.402 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:32.402 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:32.402 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:32.402 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:32.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc033820 con 0x7f63e0018390 2021-03-09T10:50:32.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.392+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:32.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.392+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:50:32.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.394+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065450 con 0x7f63e0018390 2021-03-09T10:50:32.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.775+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:32.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:32.775+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:33.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:33.775+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:33.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:33.775+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:34.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:34.775+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:34.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:34.775+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:35.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:35.760+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:35.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:35.776+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:35.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:35.776+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:36.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:36.776+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:36.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:36.776+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:37.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.391+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:37.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.391+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:37.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.391+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20180 cs=0 l=1 rev1=1 rx=0x7f63e0028390 tx=0x7f63fc066f10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:37.395 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.391+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20180 cs=0 l=1 rev1=1 rx=0x7f63e0028390 tx=0x7f63fc066f10).stop 2021-03-09T10:50:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.391+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.391+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.391+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.391+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:37.396 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:50:37.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:37.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:37.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:37.397 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:37.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:37.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:37.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:37.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000ce70 con 0x7f63d8003400 2021-03-09T10:50:37.398 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:37.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021f70 con 0x7f63d8009130 2021-03-09T10:50:37.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:37.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f63d8003840 2021-03-09T10:50:37.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:37.399 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:37.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:37.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:37.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:37.400 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:50:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.392+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:37.401 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:37.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:37.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:37.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:37.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:37.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:37.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:37.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:50:37.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:50:37.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:37.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:37.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079ba0 con 0x7f63d8003840 2021-03-09T10:50:37.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:37.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:37.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0011d50 con 0x7f63d8003840 2021-03-09T10:50:37.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:50:37.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:37.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:50:37.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=20191 cs=0 l=1 rev1=1 rx=0x7f63e00369c0 tx=0x7f63fc078d00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:37.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.393+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:37.407 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:37.400+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57511 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:37.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06fc60 con 0x7f63d8003840 2021-03-09T10:50:37.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.400+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:37.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.400+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:37.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.400+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:37.409 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:37.409 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:37.409 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:37.409 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:37.409 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:37.410 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:37.410 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:37.410 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:37.410 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:37.410 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:37.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc033820 con 0x7f63d8003840 2021-03-09T10:50:37.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.400+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:37.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.400+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:50:37.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.401+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065450 con 0x7f63d8003840 2021-03-09T10:50:37.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.776+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:37.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:37.776+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:38.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:38.776+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:38.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:38.776+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:39.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:39.776+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:39.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:39.776+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:40.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:40.761+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:40.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:40.776+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:40.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:40.776+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:40.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:40.776+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:41.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:41.776+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:41.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:41.776+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:42.402 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:50:42.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:42.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=20191 cs=0 l=1 rev1=1 rx=0x7f63e00369c0 tx=0x7f63fc078d00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:42.403 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=20191 cs=0 l=1 rev1=1 rx=0x7f63e00369c0 tx=0x7f63fc078d00).stop 2021-03-09T10:50:42.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:42.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:42.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:42.404 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=CLOSED pgs=20191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:50:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:42.405 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:42.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:42.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:42.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:42.406 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ade0 con 0x7f63d8003400 2021-03-09T10:50:42.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:42.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014280 con 0x7f63d8009130 2021-03-09T10:50:42.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:42.407 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:50:42.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:42.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:42.408 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.400+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:42.409 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.401+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:42.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.401+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:42.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.401+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:42.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.401+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:50:42.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.401+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:42.410 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.401+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:42.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.401+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:42.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.401+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:42.411 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.401+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:42.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:42.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:42.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:42.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:42.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:42.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:50:42.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:50:42.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:42.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:42.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007add0 con 0x7f63e0018390 2021-03-09T10:50:42.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:42.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:42.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007a3a0 con 0x7f63e0018390 2021-03-09T10:50:42.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:50:42.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:42.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:50:42.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20206 cs=0 l=1 rev1=1 rx=0x7f63f008fff0 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:42.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.402+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:42.416 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:42.409+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57615 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:42.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.410+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0018390 2021-03-09T10:50:42.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.410+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:42.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.410+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:42.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.410+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:42.417 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:42.418 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:42.418 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:42.418 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:42.418 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:42.419 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:42.419 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:42.419 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:42.419 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:42.419 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:42.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.414+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:50:42.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.414+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:42.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.414+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:50:42.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.415+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d30 con 0x7f63e0018390 2021-03-09T10:50:42.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.777+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:42.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:42.777+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:43.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:43.777+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:43.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:43.777+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:44.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:44.777+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:44.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:44.777+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:45.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:45.762+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:45.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:45.777+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:45.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:45.777+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:46.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:46.777+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:46.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:46.777+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:47.412 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20206 cs=0 l=1 rev1=1 rx=0x7f63f008fff0 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20206 cs=0 l=1 rev1=1 rx=0x7f63f008fff0 tx=0x7f63f004dd90).stop 2021-03-09T10:50:47.413 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:47.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:47.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:47.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:47.414 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:50:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:47.415 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:47.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:47.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:47.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:50:47.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:47.416 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:50:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028840 con 0x7f63d8003840 2021-03-09T10:50:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.409+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:47.417 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:47.418 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:50:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:47.419 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.410+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:47.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:47.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:47.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:47.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:47.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:47.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:50:47.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:50:47.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:47.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:47.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8003840 2021-03-09T10:50:47.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:47.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:47.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:50:47.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:50:47.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:47.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 secure :-1 s=READY pgs=20219 cs=0 l=1 rev1=1 rx=0x7f63f4072c80 tx=0x7f63f4009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:47.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:50:47.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.411+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:47.425 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:47.417+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57713 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:47.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.418+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63d8003840 2021-03-09T10:50:47.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.418+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:47.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.418+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:47.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.418+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:47.427 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:47.427 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:47.427 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:47.427 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:47.427 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:47.428 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:47.428 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:47.428 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:47.428 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:47.428 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:47.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.418+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8003840 2021-03-09T10:50:47.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.418+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:47.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.418+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:50:47.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.419+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402dbc0 con 0x7f63d8003840 2021-03-09T10:50:47.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.777+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:47.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:47.777+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:48.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:48.777+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:48.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:48.777+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:49.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:49.777+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:49.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:49.777+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:50.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:50.763+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:50.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:50.778+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:50.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:50.778+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:50.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:50.778+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:51.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:51.778+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:51.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:51.778+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 secure :-1 s=READY pgs=20219 cs=0 l=1 rev1=1 rx=0x7f63f4072c80 tx=0x7f63f4009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:52.420 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 secure :-1 s=READY pgs=20219 cs=0 l=1 rev1=1 rx=0x7f63f4072c80 tx=0x7f63f4009990).stop 2021-03-09T10:50:52.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:52.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:52.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007afc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:52.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 unknown :-1 s=CLOSED pgs=20219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:52.421 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:50:52.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:52.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:52.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:52.422 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:52.423 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003400 2021-03-09T10:50:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002b930 con 0x7f63d8009130 2021-03-09T10:50:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:52.424 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007c350 con 0x7f63e0018390 2021-03-09T10:50:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:52.425 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.417+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.418+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.418+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:50:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.418+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:52.426 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.418+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:52.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:52.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:52.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:50:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:50:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:52.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:50:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:50:52.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:50:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.419+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.420+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20237 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f4046e80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.420+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:50:52.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.420+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:52.433 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:52.424+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57814 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T10:50:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.425+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.425+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:52.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.425+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:52.435 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:52.436 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:52.437 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:52.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T10:50:52.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.431+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:52.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.431+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:50:52.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.431+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005c10 con 0x7f63e0018390 2021-03-09T10:50:52.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.778+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:52.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:52.778+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:53.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:53.778+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:53.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:53.778+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:54.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:54.778+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:54.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:54.778+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:55.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:55.764+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:50:55.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:55.778+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:55.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:55.778+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:56.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:56.778+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:56.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:56.778+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:57.427 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:50:57.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:50:57.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20237 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f4046e80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:50:57.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20237 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f4046e80).stop 2021-03-09T10:50:57.428 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:57.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:50:57.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:50:57.429 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:50:57.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:50:57.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:57.430 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:50:57.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:57.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:50:57.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:50:57.431 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:50:57.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:57.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:50:57.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:57.432 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:50:57.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:50:57.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63d8003840 2021-03-09T10:50:57.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:50:57.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:50:57.433 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:57.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:50:57.434 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.425+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:50:57.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.426+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:50:57.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.426+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:50:57.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.426+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:50:57.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.426+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:50:57.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.426+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:50:57.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.426+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:50:57.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.426+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:50:57.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:50:57.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:57.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:57.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:50:57.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:50:57.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:50:57.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:50:57.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:50:57.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:57.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8003840 2021-03-09T10:50:57.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:50:57.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:50:57.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.427+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:50:57.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.428+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:50:57.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.428+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:57.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.428+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 secure :-1 s=READY pgs=20253 cs=0 l=1 rev1=1 rx=0x7f63f406b040 tx=0x7f63f40051c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:50:57.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.429+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:50:57.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.429+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:50:57.441 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:50:57.432+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 57912 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:50:57.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40341b0 con 0x7f63d8003840 2021-03-09T10:50:57.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.433+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:50:57.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.433+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:50:57.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.433+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:50:57.443 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:50:57.443 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:50:57.443 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:57.444 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:50:57.444 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:50:57.444 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:50:57.444 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:50:57.445 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:50:57.445 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:50:57.445 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:50:57.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.436+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40341b0 con 0x7f63d8003840 2021-03-09T10:50:57.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.436+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:50:57.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.436+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:50:57.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.438+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40532c0 con 0x7f63d8003840 2021-03-09T10:50:57.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.779+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:57.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:57.779+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:58.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:58.779+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:58.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:58.779+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:50:59.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:59.779+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:50:59.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:50:59.779+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:00.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:00.765+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:00.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:00.779+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:00.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:00.779+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:00.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:00.779+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:01.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:01.779+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:01.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:01.779+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:02.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:02.435 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 secure :-1 s=READY pgs=20253 cs=0 l=1 rev1=1 rx=0x7f63f406b040 tx=0x7f63f40051c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:02.436 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 secure :-1 s=READY pgs=20253 cs=0 l=1 rev1=1 rx=0x7f63f406b040 tx=0x7f63f40051c0).stop 2021-03-09T10:51:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:02.437 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007afc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:02.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007afc0 unknown :-1 s=CLOSED pgs=20253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:02.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:51:02.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:02.438 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:02.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:02.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:02.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:02.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:02.439 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:02.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:51:02.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:02.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016330 con 0x7f63d8009130 2021-03-09T10:51:02.440 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:02.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63e0018390 2021-03-09T10:51:02.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:02.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:02.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:02.441 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.433+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:02.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.434+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:51:02.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.434+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:02.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.434+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:02.442 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.434+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:02.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.434+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:02.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.434+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:02.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.434+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:02.443 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.434+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:02.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:02.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:02.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:02.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:02.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:02.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:51:02.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:51:02.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:02.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:02.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0066a20 con 0x7f63e0018390 2021-03-09T10:51:02.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:02.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:02.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079ba0 con 0x7f63e0018390 2021-03-09T10:51:02.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:51:02.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.435+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:02.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.436+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:51:02.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.436+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20266 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc066ff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:02.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.436+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:02.449 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:02.441+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58013 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:02.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e0018390 2021-03-09T10:51:02.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.442+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:02.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.442+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:02.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.442+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:02.452 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:02.452 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:02.453 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:02.454 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:02.455 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:02.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.443+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e0018390 2021-03-09T10:51:02.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.443+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:02.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.443+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:51:02.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.445+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc078d70 con 0x7f63e0018390 2021-03-09T10:51:02.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.779+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:02.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:02.779+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:03.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:03.779+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:03.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:03.779+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:04.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:04.780+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:04.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:04.780+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:05.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:05.766+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:05.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:05.779+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:05.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:05.779+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:06.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:06.779+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:06.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:06.779+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:07.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:07.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:07.444 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20266 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc066ff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:07.445 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20266 cs=0 l=1 rev1=1 rx=0x7f63fc072d80 tx=0x7f63fc066ff0).stop 2021-03-09T10:51:07.446 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:07.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:07.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:07.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:07.447 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:51:07.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:07.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:07.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:07.448 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:07.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:07.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:07.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:07.449 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T10:51:07.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:07.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001230 con 0x7f63d8009130 2021-03-09T10:51:07.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:07.450 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63d8003840 2021-03-09T10:51:07.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:07.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:07.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:07.451 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:07.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:07.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:07.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:51:07.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:07.452 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.442+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:07.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.443+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:07.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.443+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:07.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.443+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:07.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:07.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:07.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:07.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:07.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:07.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:51:07.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:51:07.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:07.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:07.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:51:07.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:07.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:07.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8003840 2021-03-09T10:51:07.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:51:07.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.444+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:07.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.446+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00013b0 secure :-1 s=READY pgs=20280 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4005e20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:07.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.446+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:51:07.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.446+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:07.458 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:07.450+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58111 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:07.459 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072470 con 0x7f63d8003840 2021-03-09T10:51:07.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.450+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:07.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.450+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:07.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.450+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:07.460 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:07.460 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:07.461 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:07.461 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:07.461 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:07.462 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:07.462 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:07.462 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:07.462 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:07.463 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:07.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072290 con 0x7f63d8003840 2021-03-09T10:51:07.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.451+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:07.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.451+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:51:07.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.452+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4053b90 con 0x7f63d8003840 2021-03-09T10:51:07.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.780+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:07.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:07.780+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:08.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:08.780+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:08.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:08.780+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:09.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:09.780+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:09.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:09.780+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:10.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:10.767+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:10.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:10.780+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:10.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:10.780+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:10.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:10.780+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:11.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:11.781+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:11.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:11.781+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:12.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:12.453 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00013b0 secure :-1 s=READY pgs=20280 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4005e20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00013b0 secure :-1 s=READY pgs=20280 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4005e20).stop 2021-03-09T10:51:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:12.454 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00013b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00013b0 unknown :-1 s=CLOSED pgs=20280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:51:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:12.455 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:12.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:12.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:12.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:12.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:12.456 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:12.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:51:12.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:12.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.450+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006ce70 con 0x7f63d8009130 2021-03-09T10:51:12.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.451+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:12.457 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.451+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007ac90 con 0x7f63e0018390 2021-03-09T10:51:12.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.451+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:12.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.451+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:12.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.451+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:12.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.451+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:12.458 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.451+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:12.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.451+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:12.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:51:12.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:12.460 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:12.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:12.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:12.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:12.461 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:12.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:12.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:12.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:12.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:12.462 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:51:12.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.452+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:51:12.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:12.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:12.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ff10 con 0x7f63e0018390 2021-03-09T10:51:12.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:12.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:12.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:51:12.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:51:12.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:12.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20291 cs=0 l=1 rev1=1 rx=0x7f63f400ef40 tx=0x7f63f4072470).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:12.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:51:12.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.453+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:12.466 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:12.460+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58212 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.461+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T10:51:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.461+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.461+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.461+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:12.467 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:12.468 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:12.468 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:12.468 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:12.468 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:12.469 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:12.469 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:12.469 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:12.469 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:12.469 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:12.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.463+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T10:51:12.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.463+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:12.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.463+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:51:12.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.465+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f402c370 con 0x7f63e0018390 2021-03-09T10:51:12.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.781+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:12.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:12.781+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:13.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:13.781+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:13.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:13.781+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:14.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:14.781+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:14.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:14.781+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:15.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:15.768+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:15.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:15.781+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:15.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:15.781+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:16.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:16.781+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:16.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:16.781+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:17.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:17.463 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:17.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20291 cs=0 l=1 rev1=1 rx=0x7f63f400ef40 tx=0x7f63f4072470).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:17.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20291 cs=0 l=1 rev1=1 rx=0x7f63f400ef40 tx=0x7f63f4072470).stop 2021-03-09T10:51:17.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:17.464 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:51:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:17.465 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:17.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:17.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:17.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:17.466 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:17.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:17.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065dc0 con 0x7f63d8003400 2021-03-09T10:51:17.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:17.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f63d8009130 2021-03-09T10:51:17.467 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:17.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63d8003840 2021-03-09T10:51:17.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:17.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:17.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:17.468 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:17.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.460+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:17.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.461+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:17.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.461+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:51:17.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.461+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:17.469 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.461+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:17.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.461+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:17.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.461+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:17.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.461+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:17.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:51:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:51:17.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:17.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:17.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:51:17.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:17.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007add0 con 0x7f63d8003840 2021-03-09T10:51:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:51:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068540 secure :-1 s=READY pgs=20302 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0013b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:17.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:51:17.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.462+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:17.476 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:17.467+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58310 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:17.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:51:17.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.468+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:17.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.468+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:17.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.468+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:17.478 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:17.478 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:17.478 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:17.479 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:17.479 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:17.479 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:17.479 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:17.480 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:17.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0073340 con 0x7f63d8003840 2021-03-09T10:51:17.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.468+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:17.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.468+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:51:17.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.469+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:51:17.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.781+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:17.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:17.781+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:18.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:18.781+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:18.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:18.781+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:19.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:19.782+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:19.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:19.782+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:20.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:20.769+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:20.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:20.781+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:20.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:20.781+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:20.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:20.781+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:21.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:21.782+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:21.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:21.782+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:22.470 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:22.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:22.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068540 secure :-1 s=READY pgs=20302 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0013b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:22.471 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068540 secure :-1 s=READY pgs=20302 cs=0 l=1 rev1=1 rx=0x7f63f0015590 tx=0x7f63f0013b30).stop 2021-03-09T10:51:22.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:22.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:22.472 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:22.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068540 unknown :-1 s=CLOSED pgs=20302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:22.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:51:22.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:22.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:22.473 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:22.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:22.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:22.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:22.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:22.474 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:51:22.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:22.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079bf0 con 0x7f63d8009130 2021-03-09T10:51:22.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:22.475 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63e0018390 2021-03-09T10:51:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.468+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:22.476 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:51:22.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:22.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:22.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:22.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:22.477 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:22.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:22.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:22.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:22.478 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:22.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:22.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:22.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:22.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:51:22.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:51:22.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:22.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:22.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:51:22.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:22.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:22.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0066a20 con 0x7f63e0018390 2021-03-09T10:51:22.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:51:22.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.469+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:22.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.470+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:51:22.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.470+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:22.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.470+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20315 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:22.482 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:22.475+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58412 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:22.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:51:22.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.476+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.476+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.476+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:22.484 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:22.485 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:22.485 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:22.485 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:22.485 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:22.485 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:22.486 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:22.486 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:22.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0711d0 con 0x7f63e0018390 2021-03-09T10:51:22.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.476+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:22.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.476+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:51:22.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.477+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc051200 con 0x7f63e0018390 2021-03-09T10:51:22.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.782+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:22.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:22.782+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:23.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:23.782+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:23.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:23.782+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:24.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:24.782+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:24.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:24.782+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:25.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:25.770+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:25.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:25.782+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:25.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:25.782+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:26.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:26.783+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:26.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:26.783+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:27.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:27.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:27.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20315 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a740).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:27.479 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20315 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc04a740).stop 2021-03-09T10:51:27.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:27.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:27.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:27.480 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:51:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:27.481 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:27.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:27.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:27.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:27.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079390 con 0x7f63d8003400 2021-03-09T10:51:27.482 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:27.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:51:27.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:27.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079bf0 con 0x7f63d8003840 2021-03-09T10:51:27.483 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:27.484 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.476+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:51:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:27.485 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:27.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:27.486 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:27.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:27.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:27.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:27.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:51:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:51:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:27.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:51:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ff10 con 0x7f63d8003840 2021-03-09T10:51:27.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:51:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.477+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.478+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:51:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.478+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=20330 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f4017f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:27.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.478+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:27.491 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:27.484+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58510 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:27.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63d8003840 2021-03-09T10:51:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.485+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.485+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.485+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:27.492 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:27.493 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:27.494 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:27.494 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.488+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005110 con 0x7f63d8003840 2021-03-09T10:51:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.488+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.488+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:51:27.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.490+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63d8003840 2021-03-09T10:51:27.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.783+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:27.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:27.783+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:28.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:28.783+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:28.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:28.783+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:29.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:29.783+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:29.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:29.783+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:30.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:30.771+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:30.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:30.783+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:30.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:30.783+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:30.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:30.783+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:31.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:31.783+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:31.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:31.783+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:32.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:32.487 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:32.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=20330 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f4017f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:32.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 secure :-1 s=READY pgs=20330 cs=0 l=1 rev1=1 rx=0x7f63f4009ab0 tx=0x7f63f4017f40).stop 2021-03-09T10:51:32.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:32.488 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0001030 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0001030 unknown :-1 s=CLOSED pgs=20330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:51:32.489 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:32.490 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:32.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:32.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8003400 2021-03-09T10:51:32.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:32.491 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8009130 2021-03-09T10:51:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63e0018390 2021-03-09T10:51:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:32.492 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:32.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:32.493 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.485+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.486+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.486+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:51:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.486+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.486+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:32.495 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.486+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.486+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.486+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.486+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:32.496 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:51:32.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:51:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017490 con 0x7f63e0018390 2021-03-09T10:51:32.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63e0018390 2021-03-09T10:51:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:51:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.487+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:32.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.488+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:51:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.488+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:32.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.488+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20349 cs=0 l=1 rev1=1 rx=0x7f63f00035c0 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:32.501 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:32.493+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58611 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:32.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:51:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.494+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.494+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.494+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:32.502 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:32.503 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:32.504 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:32.504 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:32.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.501+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:51:32.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.501+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:32.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.501+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:51:32.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.501+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d30 con 0x7f63e0018390 2021-03-09T10:51:32.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.783+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:32.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:32.783+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:33.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:33.784+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:33.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:33.784+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:34.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:34.784+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:34.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:34.784+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:35.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:35.772+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:35.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:35.784+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:35.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:35.784+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:36.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:36.783+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:36.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:36.783+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:37.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:37.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:37.497 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20349 cs=0 l=1 rev1=1 rx=0x7f63f00035c0 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:37.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20349 cs=0 l=1 rev1=1 rx=0x7f63f00035c0 tx=0x7f63f0000bc0).stop 2021-03-09T10:51:37.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:37.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:37.498 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:37.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:37.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:51:37.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:37.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:37.499 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:37.500 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f63d8003400 2021-03-09T10:51:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006dd10 con 0x7f63d8009130 2021-03-09T10:51:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:37.501 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63d8003840 2021-03-09T10:51:37.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:37.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:37.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:37.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:37.502 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.494+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:37.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.495+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:37.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.495+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:51:37.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.495+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:37.503 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.495+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:37.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.495+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:37.504 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.495+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.495+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.495+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:37.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:37.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:37.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:51:37.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:51:37.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:37.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:37.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00298e0 con 0x7f63d8003840 2021-03-09T10:51:37.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:37.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:37.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017490 con 0x7f63d8003840 2021-03-09T10:51:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:51:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 secure :-1 s=READY pgs=20356 cs=0 l=1 rev1=1 rx=0x7f63f0052010 tx=0x7f63f007c430).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:37.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:51:37.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.496+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:37.509 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:37.502+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58709 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:37.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:51:37.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.502+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.502+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.502+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:37.511 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:37.512 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:37.513 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:37.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007c990 con 0x7f63d8003840 2021-03-09T10:51:37.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.502+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:37.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.502+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:51:37.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.503+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:51:37.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.784+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:37.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:37.784+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:38.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:38.784+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:38.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:38.784+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:39.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:39.784+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:39.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:39.784+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:40.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:40.773+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:40.784+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:40.784+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:40.784+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:41.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:41.784+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:41.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:41.784+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:42.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:51:42.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:42.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 secure :-1 s=READY pgs=20356 cs=0 l=1 rev1=1 rx=0x7f63f0052010 tx=0x7f63f007c430).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:42.505 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 secure :-1 s=READY pgs=20356 cs=0 l=1 rev1=1 rx=0x7f63f0052010 tx=0x7f63f007c430).stop 2021-03-09T10:51:42.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:42.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:42.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:42.506 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 unknown :-1 s=CLOSED pgs=20356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:42.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:51:42.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:42.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:42.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:42.507 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:51:42.508 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079bf0 con 0x7f63d8009130 2021-03-09T10:51:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007bb00 con 0x7f63e0018390 2021-03-09T10:51:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:42.509 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:42.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.502+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:42.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.503+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:42.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.503+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:42.510 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.503+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:42.511 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.504+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:51:42.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.504+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:42.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.504+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:42.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.504+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:42.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.504+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:42.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.504+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:42.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:42.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:42.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:42.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:42.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:42.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:51:42.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:51:42.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:42.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:42.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001cd0 con 0x7f63e0018390 2021-03-09T10:51:42.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:42.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:42.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:51:42.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:51:42.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.505+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:42.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.506+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:51:42.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.506+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20376 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f406bfa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:42.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.506+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:42.516 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:42.510+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58805 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:42.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.510+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T10:51:42.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.510+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:42.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.511+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:42.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.511+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:42.518 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:42.518 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:42.518 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:42.519 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:42.519 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:42.519 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:42.519 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:42.520 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:42.520 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:42.520 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:42.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.516+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63e0018390 2021-03-09T10:51:42.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.516+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:42.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.516+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:51:42.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.516+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63e0018390 2021-03-09T10:51:42.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.785+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:42.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:42.785+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:43.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:43.785+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:43.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:43.785+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:44.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:44.785+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:44.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:44.785+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:45.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:45.774+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:45.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:45.785+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:45.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:45.785+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:46.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:46.785+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:46.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:46.785+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:47.512 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.510+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:47.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.510+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:47.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.510+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20376 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f406bfa0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:47.513 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.510+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20376 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f406bfa0).stop 2021-03-09T10:51:47.514 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:47.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:47.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:47.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:47.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:51:47.515 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:47.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:47.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:47.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:47.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:47.516 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:47.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:47.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63d8003400 2021-03-09T10:51:47.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:47.517 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f63d8009130 2021-03-09T10:51:47.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:47.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068120 con 0x7f63d8003840 2021-03-09T10:51:47.518 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:47.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:47.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.511+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:47.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.512+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:47.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.512+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:47.519 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.512+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:47.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.512+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:51:47.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.512+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:47.520 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.512+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:47.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.513+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:47.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.513+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:47.521 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.513+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:47.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:47.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:47.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:47.522 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:47.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:47.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:51:47.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:51:47.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:47.523 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:47.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006a6e0 con 0x7f63d8003840 2021-03-09T10:51:47.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:47.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:47.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001cd0 con 0x7f63d8003840 2021-03-09T10:51:47.524 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:51:47.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:47.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:51:47.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:47.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.514+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 secure :-1 s=READY pgs=20392 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f401a270).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:47.526 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:47.519+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 58908 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:47.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005190 con 0x7f63d8003840 2021-03-09T10:51:47.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.520+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:47.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.520+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:47.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.520+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:47.528 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:47.528 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:47.528 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:47.528 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:47.529 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:47.529 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:47.529 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:47.529 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:47.530 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:47.530 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:47.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.522+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4003ad0 con 0x7f63d8003840 2021-03-09T10:51:47.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.522+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:47.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.522+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:51:47.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.524+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022a80 con 0x7f63d8003840 2021-03-09T10:51:47.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.785+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:47.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:47.785+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:48.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:48.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:48.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:48.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:49.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:49.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:49.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:49.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:50.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:50.775+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:50.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:50.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:50.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:50.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:50.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:50.786+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:51.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:51.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:51.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:51.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:52.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:51:52.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:52.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 secure :-1 s=READY pgs=20392 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f401a270).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:52.525 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 secure :-1 s=READY pgs=20392 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f401a270).stop 2021-03-09T10:51:52.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:52.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:52.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:52.526 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068410 unknown :-1 s=CLOSED pgs=20392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:52.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:51:52.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:52.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:52.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:52.527 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:52.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:52.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:52.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:52.528 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T10:51:52.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:52.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:51:52.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:52.529 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016330 con 0x7f63e0018390 2021-03-09T10:51:52.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.520+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:52.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:52.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:52.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:52.530 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:52.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:52.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:51:52.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:52.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:52.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:52.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:52.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.521+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:52.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:52.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:52.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:52.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:52.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:52.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:51:52.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:51:52.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:52.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:52.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067a00 con 0x7f63e0018390 2021-03-09T10:51:52.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:52.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:52.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:51:52.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:51:52.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:52.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20398 cs=0 l=1 rev1=1 rx=0x7f63e000c440 tx=0x7f63fc065800).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:52.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:51:52.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.522+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:52.536 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:52.528+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59007 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:52.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:51:52.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.528+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:52.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.528+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:52.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.528+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:52.538 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:52.539 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:52.539 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:52.539 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:52.539 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:52.539 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:52.540 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:52.540 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:52.540 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:52.540 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:52.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.528+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:51:52.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.528+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:52.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.528+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:51:52.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.530+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc011310 con 0x7f63e0018390 2021-03-09T10:51:52.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:52.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:52.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:53.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:53.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:53.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:53.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:54.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:54.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:54.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:54.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:55.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:55.776+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:51:55.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:55.787+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:55.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:55.787+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:56.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:56.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:56.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:56.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:57.531 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:51:57.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:51:57.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20398 cs=0 l=1 rev1=1 rx=0x7f63e000c440 tx=0x7f63fc065800).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:51:57.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20398 cs=0 l=1 rev1=1 rx=0x7f63e000c440 tx=0x7f63fc065800).stop 2021-03-09T10:51:57.532 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:57.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:51:57.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:51:57.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:51:57.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:51:57.533 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0016910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:57.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:51:57.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:57.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:51:57.534 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:51:57.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:51:57.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:57.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:51:57.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:57.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021060 con 0x7f63d8009130 2021-03-09T10:51:57.535 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:51:57.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63d8003840 2021-03-09T10:51:57.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:51:57.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:51:57.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:57.536 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:51:57.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:51:57.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0016910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:51:57.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:51:57.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:51:57.537 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:51:57.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:51:57.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:51:57.538 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.529+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:51:57.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:51:57.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:57.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:57.539 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:51:57.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:51:57.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:51:57.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:51:57.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:51:57.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:57.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005efb0 con 0x7f63d8003840 2021-03-09T10:51:57.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:51:57.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:51:57.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006a6e0 con 0x7f63d8003840 2021-03-09T10:51:57.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:51:57.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:57.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:51:57.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:51:57.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.530+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0016910 secure :-1 s=READY pgs=20412 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f400ecc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:51:57.543 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:51:57.537+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59107 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:51:57.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63d8003840 2021-03-09T10:51:57.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.538+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:51:57.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.538+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:51:57.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.538+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:51:57.545 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:51:57.545 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:51:57.545 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:57.546 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:51:57.546 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:51:57.546 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:51:57.546 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:51:57.546 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:51:57.547 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:51:57.547 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:51:57.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f401bed0 con 0x7f63d8003840 2021-03-09T10:51:57.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.538+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:51:57.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.538+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:51:57.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.539+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405c2f0 con 0x7f63d8003840 2021-03-09T10:51:57.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:57.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:57.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:58.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:58.786+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:58.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:58.786+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:51:59.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:59.787+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:51:59.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:51:59.787+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:00.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:00.777+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:00.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:00.787+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:00.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:00.787+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:00.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:00.787+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:01.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:01.787+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:01.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:01.787+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:02.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0016910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:02.540 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0016910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:02.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0016910 secure :-1 s=READY pgs=20412 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f400ecc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:02.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0016910 secure :-1 s=READY pgs=20412 cs=0 l=1 rev1=1 rx=0x7f63f400b510 tx=0x7f63f400ecc0).stop 2021-03-09T10:52:02.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:02.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:02.541 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0016910 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:02.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0016910 unknown :-1 s=CLOSED pgs=20412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:02.542 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:52:02.543 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:02.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:02.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:02.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:02.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:02.544 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:02.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:02.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:52:02.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:02.545 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028860 con 0x7f63d8009130 2021-03-09T10:52:02.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:02.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000be90 con 0x7f63e0018390 2021-03-09T10:52:02.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:02.546 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:02.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:02.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:02.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.538+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:02.547 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:02.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:52:02.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:02.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:02.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:02.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:02.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:02.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:02.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:02.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:02.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:02.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:02.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:52:02.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.539+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:52:02.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:02.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:02.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0015a60 con 0x7f63e0018390 2021-03-09T10:52:02.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:02.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:02.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005efb0 con 0x7f63e0018390 2021-03-09T10:52:02.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:52:02.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:02.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:52:02.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20425 cs=0 l=1 rev1=1 rx=0x7f63f40722d0 tx=0x7f63f4005180).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:02.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.540+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:02.553 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:02.546+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59206 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:02.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f406a4e0 con 0x7f63e0018390 2021-03-09T10:52:02.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.546+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:02.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.546+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:02.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.546+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:02.555 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:02.555 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:02.555 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:02.555 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:02.555 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:02.556 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:02.556 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:02.556 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:02.556 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:02.556 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:02.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40532c0 con 0x7f63e0018390 2021-03-09T10:52:02.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.546+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:02.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.546+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:52:02.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.547+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T10:52:02.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.788+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:02.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:02.788+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:03.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:03.788+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:03.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:03.788+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:04.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:04.788+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:04.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:04.788+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:05.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:05.778+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:05.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:05.788+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:05.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:05.788+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:06.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:06.788+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:06.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:06.788+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:07.548 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:07.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:07.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20425 cs=0 l=1 rev1=1 rx=0x7f63f40722d0 tx=0x7f63f4005180).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:07.549 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20425 cs=0 l=1 rev1=1 rx=0x7f63f40722d0 tx=0x7f63f4005180).stop 2021-03-09T10:52:07.550 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:07.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:07.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:07.551 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:07.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:52:07.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:07.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:07.552 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:07.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:07.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:07.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:07.553 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:07.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00293e0 con 0x7f63d8003400 2021-03-09T10:52:07.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:07.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017d50 con 0x7f63d8009130 2021-03-09T10:52:07.554 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:07.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63d8003840 2021-03-09T10:52:07.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.546+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:07.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:07.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:07.555 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:07.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:07.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:07.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:52:07.556 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:07.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:07.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:07.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:07.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:07.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.547+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:07.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:07.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:07.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:07.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:07.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:52:07.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:52:07.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:07.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:07.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006ce90 con 0x7f63d8003840 2021-03-09T10:52:07.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:07.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:07.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067a00 con 0x7f63d8003840 2021-03-09T10:52:07.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:52:07.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:07.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=20438 cs=0 l=1 rev1=1 rx=0x7f63e005ec90 tx=0x7f63fc052f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:07.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:52:07.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.548+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:07.562 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:07.554+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59306 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:07.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:52:07.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.554+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:07.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.554+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:07.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.554+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:07.564 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:07.565 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:07.565 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:07.565 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:07.565 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:07.566 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:07.566 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:07.566 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:07.566 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:07.567 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:07.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.554+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058aa0 con 0x7f63d8003840 2021-03-09T10:52:07.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.554+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:07.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.554+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:52:07.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.556+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:52:07.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.788+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:07.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:07.788+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:08.788+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:08.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:08.788+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:09.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:09.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:09.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:09.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:10.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:10.779+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:10.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:10.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:10.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:10.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:10.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:10.789+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:11.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:11.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:11.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:11.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:12.557 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:12.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:12.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=20438 cs=0 l=1 rev1=1 rx=0x7f63e005ec90 tx=0x7f63fc052f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:12.558 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=20438 cs=0 l=1 rev1=1 rx=0x7f63e005ec90 tx=0x7f63fc052f00).stop 2021-03-09T10:52:12.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:12.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:12.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:12.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=20438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:12.559 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:52:12.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:12.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:12.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:12.560 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:12.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:12.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:12.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:12.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021060 con 0x7f63d8003400 2021-03-09T10:52:12.561 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:52:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:52:12.562 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:12.563 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:12.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:52:12.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:12.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:12.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:12.564 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:12.565 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.555+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:12.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:12.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:12.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:12.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:12.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:12.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:52:12.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:52:12.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:12.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:12.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025520 con 0x7f63e0018390 2021-03-09T10:52:12.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:12.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:12.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0015a60 con 0x7f63e0018390 2021-03-09T10:52:12.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:52:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20453 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4003b80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:52:12.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.556+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:12.570 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:12.562+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59405 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.563+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4053eb0 con 0x7f63e0018390 2021-03-09T10:52:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.563+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.563+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:12.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.563+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:12.572 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:12.572 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:12.572 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:12.572 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:12.572 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:12.573 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:12.573 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:12.573 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:12.573 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:12.574 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.563+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4053eb0 con 0x7f63e0018390 2021-03-09T10:52:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.563+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.563+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:52:12.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.564+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63e0018390 2021-03-09T10:52:12.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.788+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:12.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:12.788+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:13.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:13.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:13.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:13.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:14.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:14.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:14.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:14.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:15.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:15.780+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:15.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:15.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:15.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:15.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:16.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:16.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:16.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:16.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:17.566 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:52:17.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:17.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20453 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4003b80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:17.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20453 cs=0 l=1 rev1=1 rx=0x7f63f4070260 tx=0x7f63f4003b80).stop 2021-03-09T10:52:17.567 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:17.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:17.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:17.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:17.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:52:17.568 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:17.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:17.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:17.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:17.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:17.569 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:17.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:17.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9a0 con 0x7f63d8003400 2021-03-09T10:52:17.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:17.570 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:52:17.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:17.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023590 con 0x7f63d8003840 2021-03-09T10:52:17.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:17.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:17.571 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:17.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:17.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:17.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:17.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:52:17.572 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:17.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.563+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:17.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.564+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:17.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.564+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:17.573 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.564+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:17.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:17.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:17.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:17.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:17.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:17.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:52:17.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:52:17.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:17.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:17.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067a00 con 0x7f63d8003840 2021-03-09T10:52:17.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:17.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:17.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00298e0 con 0x7f63d8003840 2021-03-09T10:52:17.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:52:17.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:17.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:52:17.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:17.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.565+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=20467 cs=0 l=1 rev1=1 rx=0x7f63e001c030 tx=0x7f63f004de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:17.579 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:17.571+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59505 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:17.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:52:17.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.571+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:17.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.571+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:17.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.571+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:17.580 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:17.581 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:17.581 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:17.581 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:17.581 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:17.582 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:17.582 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:17.582 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:17.582 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:17.583 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:17.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.571+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0013b30 con 0x7f63d8003840 2021-03-09T10:52:17.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.571+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:17.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.571+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:52:17.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.573+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:52:17.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:17.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:17.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:18.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:18.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:18.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:18.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:19.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:19.789+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:19.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:19.789+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:20.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:20.781+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:20.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:20.790+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:20.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:20.790+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:20.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:20.790+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:21.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:21.790+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:21.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:21.790+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:22.574 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:22.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:22.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=20467 cs=0 l=1 rev1=1 rx=0x7f63e001c030 tx=0x7f63f004de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:22.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 secure :-1 s=READY pgs=20467 cs=0 l=1 rev1=1 rx=0x7f63e001c030 tx=0x7f63f004de00).stop 2021-03-09T10:52:22.575 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:22.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:22.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001c150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:22.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001c150 unknown :-1 s=CLOSED pgs=20467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:22.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:52:22.576 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:22.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:22.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:22.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:22.577 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:22.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:22.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:22.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022750 con 0x7f63d8003400 2021-03-09T10:52:22.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:22.578 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069670 con 0x7f63d8009130 2021-03-09T10:52:22.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:22.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63e0018390 2021-03-09T10:52:22.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:22.579 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:22.580 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:52:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:22.581 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.572+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:22.582 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:22.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:52:22.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:52:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:52:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:22.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:22.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006ce90 con 0x7f63e0018390 2021-03-09T10:52:22.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:52:22.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.573+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:22.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.574+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:52:22.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.574+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:22.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.574+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20480 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc0488e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:22.587 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:22.579+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59604 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.580+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058cb0 con 0x7f63e0018390 2021-03-09T10:52:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.580+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.580+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:22.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.580+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:22.589 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:22.590 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:22.590 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:22.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.583+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e0018390 2021-03-09T10:52:22.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.583+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:22.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.583+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:52:22.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.584+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:52:22.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.790+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:22.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:22.790+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:23.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:23.790+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:23.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:23.790+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:24.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:24.791+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:24.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:24.791+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:25.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:25.782+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:25.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:25.790+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:25.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:25.790+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:26.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:26.791+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:26.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:26.791+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:27.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:27.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:27.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20480 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc0488e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:27.583 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20480 cs=0 l=1 rev1=1 rx=0x7f63fc01f9b0 tx=0x7f63fc0488e0).stop 2021-03-09T10:52:27.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:27.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:27.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:27.584 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:27.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:52:27.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:27.585 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:27.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:27.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:27.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:27.586 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017d50 con 0x7f63d8003400 2021-03-09T10:52:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:52:27.587 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016330 con 0x7f63d8003840 2021-03-09T10:52:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:27.588 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:52:27.589 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.580+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:27.590 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:27.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:27.591 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:27.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:27.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:27.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:52:27.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:52:27.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:27.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:27.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002bb10 con 0x7f63d8003840 2021-03-09T10:52:27.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:27.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:27.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067a00 con 0x7f63d8003840 2021-03-09T10:52:27.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:52:27.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:27.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:52:27.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 secure :-1 s=READY pgs=20494 cs=0 l=1 rev1=1 rx=0x7f63f00668c0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:27.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.581+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:27.595 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:27.588+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59704 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:27.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:52:27.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.588+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:27.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.588+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.588+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:27.597 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:27.598 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:27.598 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:27.598 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:27.599 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:27.599 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:27.599 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:27.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.588+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:52:27.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.588+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:27.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.588+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:52:27.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.589+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ed20 con 0x7f63d8003840 2021-03-09T10:52:27.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.791+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:27.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:27.791+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:28.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:28.791+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:28.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:28.791+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:29.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:29.791+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:29.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:29.791+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:30.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:30.783+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:30.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:30.791+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:30.791+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:30.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:30.791+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:31.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:31.792+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:31.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:31.792+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:32.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00273d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:32.592 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00273d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 secure :-1 s=READY pgs=20494 cs=0 l=1 rev1=1 rx=0x7f63f00668c0 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 secure :-1 s=READY pgs=20494 cs=0 l=1 rev1=1 rx=0x7f63f00668c0 tx=0x7f63f004db20).stop 2021-03-09T10:52:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:32.593 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00273d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:32.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 unknown :-1 s=CLOSED pgs=20494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:32.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:52:32.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:32.594 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:32.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:32.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:32.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:32.595 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:32.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:32.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005d9a0 con 0x7f63d8003400 2021-03-09T10:52:32.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:32.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f63d8009130 2021-03-09T10:52:32.596 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00253e0 con 0x7f63e0018390 2021-03-09T10:52:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:32.597 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:32.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:32.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:32.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:52:32.598 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:32.599 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:32.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:32.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:32.600 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:52:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:52:32.601 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:32.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:32.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067a00 con 0x7f63e0018390 2021-03-09T10:52:32.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:32.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:32.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025520 con 0x7f63e0018390 2021-03-09T10:52:32.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:52:32.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.589+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:32.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20509 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f405c360).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:32.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:52:32.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:32.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:32.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:32.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20509 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f405c360).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:32.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20509 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f405c360).stop 2021-03-09T10:52:32.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:32.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:32.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:32.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:32.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:52:32.609 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:32.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:32.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:32.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:32.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:32.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:32.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:32.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003400 2021-03-09T10:52:32.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:32.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63d8009130 2021-03-09T10:52:32.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:32.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013160 con 0x7f63d8003840 2021-03-09T10:52:32.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:32.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:32.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.591+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:32.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.592+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:32.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.592+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:32.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.592+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:32.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.592+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:52:32.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.592+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:32.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.592+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:32.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:32.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:32.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:32.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:32.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:32.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:32.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:32.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:32.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:52:32.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:52:32.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:32.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:32.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016fc0 con 0x7f63d8003840 2021-03-09T10:52:32.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:32.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:32.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002bb10 con 0x7f63d8003840 2021-03-09T10:52:32.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:52:32.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.593+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:32.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.594+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:52:32.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.594+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:32.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.594+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 secure :-1 s=READY pgs=20519 cs=0 l=1 rev1=1 rx=0x7f63f0011aa0 tx=0x7f63f0031fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:32.619 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:32.598+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59803 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:32.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.599+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:52:32.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.599+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:32.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.599+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:32.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.599+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:32.621 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:32.621 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:32.621 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:32.622 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:32.622 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:32.622 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:32.622 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:32.623 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:32.623 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:32.623 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:32.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.603+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:52:32.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.603+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:32.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.603+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:52:32.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.604+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0011da0 con 0x7f63d8003840 2021-03-09T10:52:32.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.792+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:32.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:32.792+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:33.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:33.792+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:33.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:33.792+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:34.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:34.791+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:34.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:34.791+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:35.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:35.784+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:35.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:35.792+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:35.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:35.792+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:36.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:36.792+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:36.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:36.792+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:37.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.599+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00273d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:37.602 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.599+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00273d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:37.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.599+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 secure :-1 s=READY pgs=20519 cs=0 l=1 rev1=1 rx=0x7f63f0011aa0 tx=0x7f63f0031fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:37.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.599+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 secure :-1 s=READY pgs=20519 cs=0 l=1 rev1=1 rx=0x7f63f0011aa0 tx=0x7f63f0031fb0).stop 2021-03-09T10:52:37.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:37.603 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00273d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00273d0 unknown :-1 s=CLOSED pgs=20519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:52:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:37.604 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:37.605 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:52:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e390 con 0x7f63d8009130 2021-03-09T10:52:37.606 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:37.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0016330 con 0x7f63e0018390 2021-03-09T10:52:37.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:37.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:37.607 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:37.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:37.608 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.600+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.601+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:52:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.601+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.601+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:37.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.601+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:37.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.601+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:37.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.601+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:37.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:37.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:37.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:37.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:37.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:37.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:52:37.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:52:37.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:37.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:37.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T10:52:37.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:37.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:37.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067a00 con 0x7f63e0018390 2021-03-09T10:52:37.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:52:37.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.602+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:37.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.603+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:52:37.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.603+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:37.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.603+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20529 cs=0 l=1 rev1=1 rx=0x7f63f4005150 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:37.615 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:37.607+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59903 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:37.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.607+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4062340 con 0x7f63e0018390 2021-03-09T10:52:37.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.607+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:37.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.607+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:37.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.607+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:37.617 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:37.617 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:37.617 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:37.618 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:37.618 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:37.618 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:37.618 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:37.619 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:37.619 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:37.619 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:37.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.609+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4062340 con 0x7f63e0018390 2021-03-09T10:52:37.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.609+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:37.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.609+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:52:37.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.610+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0018390 2021-03-09T10:52:37.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.792+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:37.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:37.792+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:38.792+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:38.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:38.792+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:39.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:39.793+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:39.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:39.793+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:40.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:40.785+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:40.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:40.793+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:40.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:40.793+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:40.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:40.793+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:41.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:41.793+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:41.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:41.793+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:42.610 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20529 cs=0 l=1 rev1=1 rx=0x7f63f4005150 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:42.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20529 cs=0 l=1 rev1=1 rx=0x7f63f4005150 tx=0x7f63f4017ed0).stop 2021-03-09T10:52:42.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:42.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:42.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:42.611 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:42.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:52:42.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:42.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:42.612 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:42.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:42.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:42.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:42.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:42.613 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007bb00 con 0x7f63d8003400 2021-03-09T10:52:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f63d8009130 2021-03-09T10:52:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:42.614 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8003840 2021-03-09T10:52:42.615 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:42.616 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:42.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:42.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:42.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.608+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:42.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:42.617 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:52:42.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:42.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:42.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:42.618 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:42.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:42.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:42.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:42.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:42.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:42.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:42.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:52:42.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:52:42.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:42.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:42.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025250 con 0x7f63d8003840 2021-03-09T10:52:42.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:42.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:42.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016fc0 con 0x7f63d8003840 2021-03-09T10:52:42.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:52:42.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.609+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:42.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.610+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:52:42.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.610+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:42.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.610+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 secure :-1 s=READY pgs=20536 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:42.623 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:42.616+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 59991 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:52:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.616+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:42.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.616+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.616+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:42.625 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:42.626 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:42.626 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:42.626 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:42.626 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:42.626 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:42.627 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:42.627 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:42.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.616+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007ed20 con 0x7f63d8003840 2021-03-09T10:52:42.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.617+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:42.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.617+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:52:42.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.618+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63d8003840 2021-03-09T10:52:42.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.793+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:42.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:42.793+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:43.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:43.793+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:43.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:43.793+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:44.793+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:44.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:44.793+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:45.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:45.786+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:45.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:45.793+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:45.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:45.793+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:46.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:46.794+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:46.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:46.794+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:47.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:52:47.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:47.619 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 secure :-1 s=READY pgs=20536 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:47.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 secure :-1 s=READY pgs=20536 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f0000bc0).stop 2021-03-09T10:52:47.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:47.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:47.620 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002f410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:47.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002f410 unknown :-1 s=CLOSED pgs=20536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:47.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:52:47.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:47.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:47.621 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:47.622 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013160 con 0x7f63d8003400 2021-03-09T10:52:47.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:47.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065590 con 0x7f63d8009130 2021-03-09T10:52:47.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:47.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0026730 con 0x7f63e0018390 2021-03-09T10:52:47.623 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:47.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:47.624 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:47.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:47.625 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.617+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:47.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.618+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:47.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.618+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:52:47.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.618+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:47.626 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.618+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:47.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:47.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:47.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:47.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:47.627 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:52:47.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:52:47.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:47.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:47.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079170 con 0x7f63e0018390 2021-03-09T10:52:47.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:47.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:47.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T10:52:47.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:52:47.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.619+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:47.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.620+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20556 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f405c440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:47.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.620+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:52:47.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.620+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:47.631 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:47.625+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60103 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:47.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.625+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f400ee80 con 0x7f63e0018390 2021-03-09T10:52:47.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.625+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:47.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.625+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:47.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.625+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:47.633 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:47.633 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:47.633 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:47.634 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:47.634 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:47.634 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:47.634 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:47.634 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:47.634 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:47.635 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:47.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.628+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f400ee80 con 0x7f63e0018390 2021-03-09T10:52:47.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.628+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:47.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.628+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:52:47.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.629+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63e0018390 2021-03-09T10:52:47.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.794+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:47.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:47.794+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:48.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:48.794+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:48.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:48.794+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:49.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:49.794+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:49.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:49.794+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:50.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:50.787+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:50.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:50.794+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:50.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:50.794+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:50.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:50.794+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:51.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:51.794+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:51.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:51.794+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:52.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:52.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:52.628 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20556 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f405c440).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20556 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f405c440).stop 2021-03-09T10:52:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:52.629 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:52.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:52.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:52:52.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.626+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:52.630 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:52.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:52.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:52.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:52.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:52.631 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00118b0 con 0x7f63d8003400 2021-03-09T10:52:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8009130 2021-03-09T10:52:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:52.632 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f63d8003840 2021-03-09T10:52:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:52.633 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.627+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:52.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.628+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:52.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.628+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:52:52.635 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.628+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:52.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.628+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:52.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:52.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:52.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:52.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:52.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:52.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:52.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:52.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:52.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:52:52.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:52:52.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:52.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:52.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002bb10 con 0x7f63d8003840 2021-03-09T10:52:52.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:52.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:52.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079170 con 0x7f63d8003840 2021-03-09T10:52:52.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:52:52.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:52.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:52:52.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:52.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.630+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065fe0 secure :-1 s=READY pgs=20572 cs=0 l=1 rev1=1 rx=0x7f63f404c040 tx=0x7f63f4046e10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:52.641 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:52.634+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60199 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:52.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40341b0 con 0x7f63d8003840 2021-03-09T10:52:52.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.635+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:52.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.635+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:52.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.635+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:52.642 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:52.643 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:52.643 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:52.643 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:52.643 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:52.643 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:52.644 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:52.644 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:52.644 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:52.644 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:52.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.638+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40341b0 con 0x7f63d8003840 2021-03-09T10:52:52.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.638+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:52.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.638+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:52:52.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.638+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400eae0 con 0x7f63d8003840 2021-03-09T10:52:52.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.794+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:52.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:52.794+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:53.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:53.794+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:53.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:53.794+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:54.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:54.795+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:54.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:54.795+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:55.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:55.788+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:52:55.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:55.795+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:55.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:55.795+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:56.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:56.795+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:56.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:56.795+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:57.636 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.634+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:52:57.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.634+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:52:57.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.634+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065fe0 secure :-1 s=READY pgs=20572 cs=0 l=1 rev1=1 rx=0x7f63f404c040 tx=0x7f63f4046e10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:52:57.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.634+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065fe0 secure :-1 s=READY pgs=20572 cs=0 l=1 rev1=1 rx=0x7f63f404c040 tx=0x7f63f4046e10).stop 2021-03-09T10:52:57.637 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:57.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:52:57.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065fe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:52:57.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065fe0 unknown :-1 s=CLOSED pgs=20572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:52:57.638 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:52:57.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:57.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:52:57.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:57.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:52:57.639 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:52:57.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:52:57.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:57.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00166d0 con 0x7f63d8003400 2021-03-09T10:52:57.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:57.640 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b260 con 0x7f63d8009130 2021-03-09T10:52:57.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:52:57.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:52:57.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:52:57.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:52:57.641 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:57.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:52:57.642 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.635+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:52:57.643 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.636+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:52:57.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.636+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:52:57.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.636+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:52:57.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.636+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:52:57.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.636+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:52:57.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.636+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:52:57.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.636+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:52:57.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:52:57.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:57.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:57.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:52:57.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:52:57.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:52:57.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:52:57.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:52:57.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:57.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006be40 con 0x7f63e0018390 2021-03-09T10:52:57.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:52:57.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:52:57.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025250 con 0x7f63e0018390 2021-03-09T10:52:57.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:52:57.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.637+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:57.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.638+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:52:57.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.638+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20584 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0023f40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:52:57.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.638+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:52:57.649 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:52:57.643+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60303 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:52:57.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.644+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63e0018390 2021-03-09T10:52:57.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.644+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:52:57.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.644+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:52:57.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.644+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:52:57.651 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:52:57.651 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:52:57.651 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:57.651 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:52:57.651 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:52:57.652 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:52:57.652 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:52:57.652 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:52:57.652 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:52:57.652 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:52:57.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.648+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63e0018390 2021-03-09T10:52:57.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.648+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:52:57.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.648+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:52:57.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.650+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ed20 con 0x7f63e0018390 2021-03-09T10:52:57.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.795+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:57.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:57.795+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:58.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:58.795+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:58.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:58.795+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:52:59.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:59.795+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:52:59.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:52:59.795+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:00.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:00.789+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:00.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:00.795+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:00.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:00.795+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:00.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:00.795+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:01.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:01.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:01.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:01.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:02.644 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:02.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:02.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20584 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0023f40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:02.645 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20584 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0023f40).stop 2021-03-09T10:53:02.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:02.646 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:02.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:02.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:02.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:53:02.647 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:02.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:02.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:02.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:02.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:02.648 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:02.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:02.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013160 con 0x7f63d8003400 2021-03-09T10:53:02.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:02.649 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002a3a0 con 0x7f63d8009130 2021-03-09T10:53:02.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:02.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:53:02.650 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.643+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.644+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.644+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.644+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.644+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:02.651 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.644+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.644+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:53:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.644+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.644+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:02.652 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.645+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:02.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.645+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:02.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.645+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:02.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:02.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:02.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:02.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:02.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:02.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:53:02.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:53:02.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:02.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:02.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8003840 2021-03-09T10:53:02.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:02.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:02.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:53:02.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:53:02.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:02.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068120 secure :-1 s=READY pgs=20594 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:02.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:53:02.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.646+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:02.659 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:02.651+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60401 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:02.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065790 con 0x7f63d8003840 2021-03-09T10:53:02.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.652+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:02.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.652+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:02.660 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.652+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:02.661 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:02.661 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:02.661 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:02.661 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:02.661 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:02.662 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:02.662 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:02.662 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:02.662 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:02.663 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:02.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065790 con 0x7f63d8003840 2021-03-09T10:53:02.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.652+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:02.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.653+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:53:02.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.654+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc066c60 con 0x7f63d8003840 2021-03-09T10:53:02.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:02.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:02.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:03.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:03.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:03.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:03.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:04.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:04.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:04.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:04.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:05.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:05.790+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:05.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:05.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:05.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:05.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:06.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:06.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:06.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:06.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:07.653 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:07.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:07.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068120 secure :-1 s=READY pgs=20594 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:07.654 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068120 secure :-1 s=READY pgs=20594 cs=0 l=1 rev1=1 rx=0x7f63e0069cc0 tx=0x7f63fc009990).stop 2021-03-09T10:53:07.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:07.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:07.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:07.655 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068120 unknown :-1 s=CLOSED pgs=20594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:07.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:53:07.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:07.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:07.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:07.656 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:07.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:07.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:07.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:07.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00166d0 con 0x7f63d8003400 2021-03-09T10:53:07.657 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000c550 con 0x7f63d8009130 2021-03-09T10:53:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:53:07.658 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:07.659 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.652+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.653+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:07.661 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:53:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:07.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:07.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:53:07.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:53:07.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:07.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:07.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d210 con 0x7f63e0018390 2021-03-09T10:53:07.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:07.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:07.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.654+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006be40 con 0x7f63e0018390 2021-03-09T10:53:07.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.655+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:53:07.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.655+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:07.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.655+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20608 cs=0 l=1 rev1=1 rx=0x7f63e0037ed0 tx=0x7f63f0031f00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:07.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.655+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:53:07.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.655+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:07.667 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:07.660+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60502 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:07.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63e0018390 2021-03-09T10:53:07.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.661+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:07.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.661+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:07.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.661+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:07.669 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:07.670 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:07.670 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:07.670 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:07.671 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.662+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63e0018390 2021-03-09T10:53:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.662+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:07.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.662+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:53:07.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.663+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ed20 con 0x7f63e0018390 2021-03-09T10:53:07.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:07.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:07.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:08.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:08.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:08.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:08.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:09.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:09.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:09.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:09.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:10.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:10.791+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:10.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:10.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:10.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:10.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:10.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:10.797+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:11.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:11.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:11.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:11.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:12.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:12.662 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:12.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20608 cs=0 l=1 rev1=1 rx=0x7f63e0037ed0 tx=0x7f63f0031f00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:12.663 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20608 cs=0 l=1 rev1=1 rx=0x7f63e0037ed0 tx=0x7f63f0031f00).stop 2021-03-09T10:53:12.664 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:12.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:12.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:12.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:12.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:53:12.665 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:12.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:12.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:12.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:12.666 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:12.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:12.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:12.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b260 con 0x7f63d8003400 2021-03-09T10:53:12.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:12.667 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00118b0 con 0x7f63d8009130 2021-03-09T10:53:12.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:12.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069510 con 0x7f63d8003840 2021-03-09T10:53:12.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:12.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:12.668 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:12.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:12.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:12.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:12.669 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:53:12.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:12.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.661+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:12.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:12.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:12.670 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:12.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:12.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:12.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:12.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:12.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:12.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:53:12.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:53:12.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:12.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:12.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d9a0 con 0x7f63d8003840 2021-03-09T10:53:12.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:12.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:12.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d210 con 0x7f63d8003840 2021-03-09T10:53:12.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:53:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.662+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.663+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:53:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.663+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067bb0 secure :-1 s=READY pgs=20618 cs=0 l=1 rev1=1 rx=0x7f63e001b3f0 tx=0x7f63f0006ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:12.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.663+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:12.675 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:12.669+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60600 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:12.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.670+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0031d70 con 0x7f63d8003840 2021-03-09T10:53:12.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.670+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:12.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.670+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:12.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.670+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:12.676 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:12.676 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:12.677 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:12.677 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:12.677 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:12.677 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:12.678 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:12.678 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:12.678 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:12.678 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:12.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.673+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0031d70 con 0x7f63d8003840 2021-03-09T10:53:12.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.673+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:12.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.673+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:53:12.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.675+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0032c30 con 0x7f63d8003840 2021-03-09T10:53:12.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:12.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:12.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:13.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:13.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:13.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:13.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:14.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:14.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:14.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:14.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:15.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:15.791+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:15.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:15.796+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:15.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:15.796+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:16.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:16.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:16.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:16.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:17.671 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.668+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:17.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.668+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:17.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067bb0 secure :-1 s=READY pgs=20618 cs=0 l=1 rev1=1 rx=0x7f63e001b3f0 tx=0x7f63f0006ca0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:17.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.668+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067bb0 secure :-1 s=READY pgs=20618 cs=0 l=1 rev1=1 rx=0x7f63e001b3f0 tx=0x7f63f0006ca0).stop 2021-03-09T10:53:17.672 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:17.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:17.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0067bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:17.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0067bb0 unknown :-1 s=CLOSED pgs=20618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:17.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:53:17.673 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:17.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:17.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:17.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:17.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:17.674 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:17.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:17.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066a20 con 0x7f63d8003400 2021-03-09T10:53:17.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:17.675 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001bcc0 con 0x7f63d8009130 2021-03-09T10:53:17.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:17.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d790 con 0x7f63e0018390 2021-03-09T10:53:17.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:17.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:17.676 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:17.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:17.677 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.669+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:17.678 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.670+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:17.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.670+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:53:17.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.670+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:17.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.670+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:17.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.670+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:17.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.670+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:17.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.670+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:17.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.671+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:17.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.671+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:17.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:17.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:17.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:17.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:53:17.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:53:17.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:17.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:17.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001a6e0 con 0x7f63e0018390 2021-03-09T10:53:17.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:17.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:17.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:53:17.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:53:17.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:17.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20639 cs=0 l=1 rev1=1 rx=0x7f63e0022510 tx=0x7f63fc065790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:17.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:53:17.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.672+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:17.684 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:17.676+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60689 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:17.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:53:17.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.677+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:17.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.677+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:17.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.677+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:17.686 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:17.686 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:17.686 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:17.687 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:17.688 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:17.688 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:17.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.681+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:53:17.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.681+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.681+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:53:17.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.683+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:53:17.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:17.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:17.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:18.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:18.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:18.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:18.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:19.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:19.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:19.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:19.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:20.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:20.792+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:20.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:20.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:20.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:20.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:20.797+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:21.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:21.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:21.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:21.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:22.679 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:22.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:22.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20639 cs=0 l=1 rev1=1 rx=0x7f63e0022510 tx=0x7f63fc065790).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:22.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 secure :-1 s=READY pgs=20639 cs=0 l=1 rev1=1 rx=0x7f63e0022510 tx=0x7f63fc065790).stop 2021-03-09T10:53:22.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:22.680 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064240 unknown :-1 s=CLOSED pgs=20639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:53:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:22.681 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:22.682 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:22.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:22.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011ab0 con 0x7f63d8003400 2021-03-09T10:53:22.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:22.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:53:22.683 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:22.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b260 con 0x7f63d8003840 2021-03-09T10:53:22.684 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.677+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:22.685 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.678+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.678+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.678+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.678+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:53:22.686 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.678+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.678+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.679+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.679+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.679+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:22.687 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0067130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:22.688 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:53:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:53:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:22.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016fc0 con 0x7f63d8003840 2021-03-09T10:53:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002bb10 con 0x7f63d8003840 2021-03-09T10:53:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:53:22.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068710 secure :-1 s=READY pgs=20653 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f406a550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:53:22.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.680+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:22.692 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:22.685+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60799 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:22.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.685+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63d8003840 2021-03-09T10:53:22.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.685+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:22.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.685+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:22.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.685+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:22.693 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:22.693 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:22.694 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:22.695 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:22.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.686+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63d8003840 2021-03-09T10:53:22.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.686+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:22.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.686+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:53:22.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.687+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4029d60 con 0x7f63d8003840 2021-03-09T10:53:22.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:22.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:22.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:23.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:23.797+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:23.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:23.797+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:24.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:24.798+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:24.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:24.798+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:25.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:25.793+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:25.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:25.798+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:25.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:25.798+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:26.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:26.798+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:26.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:26.798+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:27.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:27.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:27.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068710 secure :-1 s=READY pgs=20653 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f406a550).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:27.689 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068710 secure :-1 s=READY pgs=20653 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f406a550).stop 2021-03-09T10:53:27.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:27.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:27.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0068710 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:27.690 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0068710 unknown :-1 s=CLOSED pgs=20653 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:27.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:53:27.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:27.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:27.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:27.691 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:27.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:27.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:27.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:27.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066a20 con 0x7f63d8003400 2021-03-09T10:53:27.692 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:27.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007bb50 con 0x7f63d8009130 2021-03-09T10:53:27.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:27.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025630 con 0x7f63e0018390 2021-03-09T10:53:27.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:27.693 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:27.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:27.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:27.694 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.686+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:27.695 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.687+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:27.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.687+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:53:27.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.687+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:27.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.687+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:27.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.687+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:27.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.687+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:27.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.687+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:27.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:27.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:27.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:27.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:27.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:27.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:53:27.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:53:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002bb10 con 0x7f63e0018390 2021-03-09T10:53:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:27.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d9a0 con 0x7f63e0018390 2021-03-09T10:53:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:53:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.688+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.689+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067130 secure :-1 s=READY pgs=20665 cs=0 l=1 rev1=1 rx=0x7f63f00668e0 tx=0x7f63f0013c10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:27.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.689+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:53:27.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.689+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:27.701 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:27.693+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60900 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0039330 con 0x7f63e0018390 2021-03-09T10:53:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.694+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:27.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.694+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.694+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:27.703 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:27.704 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:27.704 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:27.704 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:27.704 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:27.704 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:27.705 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.697+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0039330 con 0x7f63e0018390 2021-03-09T10:53:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.697+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.698+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ed20 con 0x7f63e0018390 2021-03-09T10:53:27.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.698+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:53:27.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.798+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:27.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:27.798+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:28.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:28.798+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:28.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:28.798+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:29.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:29.798+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:29.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:29.798+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:30.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:30.794+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:30.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:30.798+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:30.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:30.798+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:30.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:30.798+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:31.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:31.799+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:31.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:31.799+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:32.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0067130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:32.696 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0067130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:32.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067130 secure :-1 s=READY pgs=20665 cs=0 l=1 rev1=1 rx=0x7f63f00668e0 tx=0x7f63f0013c10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:32.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067130 secure :-1 s=READY pgs=20665 cs=0 l=1 rev1=1 rx=0x7f63f00668e0 tx=0x7f63f0013c10).stop 2021-03-09T10:53:32.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:32.697 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:32.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0067130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:32.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0067130 unknown :-1 s=CLOSED pgs=20665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:32.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:53:32.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:32.698 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:32.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:32.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:32.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:32.699 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:32.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:32.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005f170 con 0x7f63d8003400 2021-03-09T10:53:32.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:32.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0069510 con 0x7f63d8009130 2021-03-09T10:53:32.700 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:32.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:53:32.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:32.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:32.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:32.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:32.701 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.694+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:32.702 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.695+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:32.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.695+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:53:32.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.695+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:32.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.695+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:32.703 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.696+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.696+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.696+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:32.704 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:53:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:53:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:32.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00118b0 con 0x7f63d8003840 2021-03-09T10:53:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002bb10 con 0x7f63d8003840 2021-03-09T10:53:32.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:53:32.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:32.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.697+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e6d0 secure :-1 s=READY pgs=20683 cs=0 l=1 rev1=1 rx=0x7f63e002fb70 tx=0x7f63f0090530).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:32.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.698+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:53:32.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.698+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:32.708 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:32.701+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 60998 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.702+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00907d0 con 0x7f63d8003840 2021-03-09T10:53:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.702+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.702+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.702+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:32.709 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:32.710 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:32.710 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:32.710 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:32.710 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:32.710 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:32.711 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:32.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.705+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00907d0 con 0x7f63d8003840 2021-03-09T10:53:32.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.705+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:32.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.705+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:53:32.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.707+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63d8003840 2021-03-09T10:53:32.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.799+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:32.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:32.799+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:33.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:33.799+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:33.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:33.799+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:34.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:34.799+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:34.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:34.799+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:35.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:35.795+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:35.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:35.799+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:35.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:35.799+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:36.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:36.799+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:36.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:36.799+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:37.705 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:37.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:37.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e6d0 secure :-1 s=READY pgs=20683 cs=0 l=1 rev1=1 rx=0x7f63e002fb70 tx=0x7f63f0090530).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:37.706 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e6d0 secure :-1 s=READY pgs=20683 cs=0 l=1 rev1=1 rx=0x7f63e002fb70 tx=0x7f63f0090530).stop 2021-03-09T10:53:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002e6d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002e6d0 unknown :-1 s=CLOSED pgs=20683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:37.707 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:53:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:37.708 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001950 con 0x7f63d8003400 2021-03-09T10:53:37.709 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d790 con 0x7f63d8009130 2021-03-09T10:53:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:37.710 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63e0018390 2021-03-09T10:53:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:37.711 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:37.712 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.703+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:37.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.704+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:37.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.704+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:53:37.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.704+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:37.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.704+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:37.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.704+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:37.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.704+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:37.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.704+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:37.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:37.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:37.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:37.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:37.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:53:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:53:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:37.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63e0018390 2021-03-09T10:53:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016fc0 con 0x7f63e0018390 2021-03-09T10:53:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:53:37.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.705+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:37.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.706+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20693 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003b40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:37.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.706+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:53:37.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.706+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:37.719 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:37.710+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61099 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:37.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.711+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T10:53:37.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.711+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:37.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.711+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:37.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.711+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:37.721 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:37.721 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:37.721 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:37.722 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:37.722 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:37.722 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:37.722 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:37.722 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:37.723 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:37.723 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:37.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.712+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4006a90 con 0x7f63e0018390 2021-03-09T10:53:37.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.712+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:37.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.712+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:53:37.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.712+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63e0018390 2021-03-09T10:53:37.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.799+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:37.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:37.799+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:38.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:38.799+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:38.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:38.799+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:39.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:39.800+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:39.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:39.800+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:40.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:40.796+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:40.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:40.799+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:40.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:40.799+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:40.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:40.800+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:41.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:41.800+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:41.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:41.800+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:42.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:42.713 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:42.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20693 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003b40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:42.714 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20693 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003b40).stop 2021-03-09T10:53:42.715 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:53:42.716 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:42.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:42.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:42.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:42.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:42.717 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:42.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:42.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027ea0 con 0x7f63d8003400 2021-03-09T10:53:42.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:42.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006d990 con 0x7f63d8009130 2021-03-09T10:53:42.718 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:42.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:53:42.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:42.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:42.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.711+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:42.719 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.712+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:42.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.712+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:42.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.712+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:42.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.712+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:53:42.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.712+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:42.720 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.712+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.712+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.712+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.712+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:42.721 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:53:42.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:53:42.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:42.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:42.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0001770 con 0x7f63d8003840 2021-03-09T10:53:42.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:42.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00118b0 con 0x7f63d8003840 2021-03-09T10:53:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:53:42.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.713+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:42.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.714+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=20705 cs=0 l=1 rev1=1 rx=0x7f63e005de90 tx=0x7f63f0003500).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:42.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.714+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:53:42.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.714+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:42.725 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:42.719+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61197 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:42.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:53:42.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.720+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:42.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.720+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:42.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.720+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:42.727 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:42.727 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:42.727 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:42.727 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:42.728 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:42.729 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:42.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.724+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:53:42.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.724+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:42.729 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.724+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:53:42.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.725+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:53:42.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.800+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:42.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:42.800+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:43.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:43.800+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:43.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:43.800+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:44.800+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:44.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:44.800+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:45.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:45.797+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:45.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:45.800+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:45.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:45.800+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:46.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:46.801+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:46.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:46.801+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:47.722 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=20705 cs=0 l=1 rev1=1 rx=0x7f63e005de90 tx=0x7f63f0003500).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=20705 cs=0 l=1 rev1=1 rx=0x7f63e005de90 tx=0x7f63f0003500).stop 2021-03-09T10:53:47.723 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:47.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:47.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:47.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=20705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:47.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:53:47.724 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:47.725 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00096f0 con 0x7f63d8003400 2021-03-09T10:53:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:47.726 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025630 con 0x7f63d8009130 2021-03-09T10:53:47.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:47.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012170 con 0x7f63e0018390 2021-03-09T10:53:47.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:47.727 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:47.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:47.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:47.728 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.720+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:47.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:47.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:53:47.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:47.730 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:47.731 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:47.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:53:47.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:53:47.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:47.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:47.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.721+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a350 con 0x7f63e0018390 2021-03-09T10:53:47.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.722+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:47.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.722+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:47.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.722+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001a6e0 con 0x7f63e0018390 2021-03-09T10:53:47.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.722+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:53:47.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.722+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:47.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.722+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:53:47.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.722+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:47.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.722+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006a980 secure :-1 s=READY pgs=20718 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc0715c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:47.736 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:47.729+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61298 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:47.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.730+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63e0018390 2021-03-09T10:53:47.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.730+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:47.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.730+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:47.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.730+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:47.737 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:47.738 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:47.738 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:47.738 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:47.738 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:47.739 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:47.739 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:47.739 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:47.739 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:47.739 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:47.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.734+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0331e0 con 0x7f63e0018390 2021-03-09T10:53:47.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.734+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:47.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.734+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:53:47.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.734+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:53:47.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.800+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:47.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:47.800+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:48.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:48.801+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:48.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:48.801+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:49.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:49.801+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:49.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:49.801+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:50.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:50.798+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:50.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:50.801+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:50.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:50.801+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:50.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:50.801+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:51.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:51.801+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:51.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:51.801+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:52.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e006a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:53:52.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e006a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:52.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006a980 secure :-1 s=READY pgs=20718 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc0715c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:52.732 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006a980 secure :-1 s=READY pgs=20718 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc0715c0).stop 2021-03-09T10:53:52.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:52.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:52.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e006a980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:52.733 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e006a980 unknown :-1 s=CLOSED pgs=20718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:52.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:53:52.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:52.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:52.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:52.734 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:53:52.735 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002c040 con 0x7f63d8009130 2021-03-09T10:53:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00195c0 con 0x7f63d8003840 2021-03-09T10:53:52.736 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.729+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:52.737 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.730+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:52.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.730+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:52.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.730+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:52.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.730+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:52.738 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.730+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:52.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.730+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:53:52.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.730+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:52.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.730+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:52.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.731+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:52.739 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.731+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:52.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.731+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:52.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.731+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:52.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:52.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:52.740 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:52.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:52.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:53:52.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:53:52.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:52.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:52.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002be90 con 0x7f63d8003840 2021-03-09T10:53:52.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:52.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:52.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0001770 con 0x7f63d8003840 2021-03-09T10:53:52.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:53:52.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.732+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:52.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.733+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=20733 cs=0 l=1 rev1=1 rx=0x7f63f008afd0 tx=0x7f63f0006bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:52.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.733+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:53:52.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.733+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:52.744 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:52.738+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61396 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:52.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00905a0 con 0x7f63d8003840 2021-03-09T10:53:52.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.738+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:52.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.738+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:52.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.738+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:52.745 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:52.745 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:52.746 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:52.746 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:52.746 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:52.746 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:52.747 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:52.747 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:52.747 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:52.747 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:52.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.743+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00905a0 con 0x7f63d8003840 2021-03-09T10:53:52.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.743+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:52.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.743+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:53:52.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.744+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:53:52.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.801+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:52.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:52.801+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:53.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:53.802+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:53.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:53.802+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:54.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:54.802+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:54.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:54.802+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:55.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:55.799+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:53:55.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:55.801+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:55.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:55.801+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:56.802+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:56.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:56.802+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:57.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:53:57.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:53:57.741 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=20733 cs=0 l=1 rev1=1 rx=0x7f63f008afd0 tx=0x7f63f0006bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:53:57.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=20733 cs=0 l=1 rev1=1 rx=0x7f63f008afd0 tx=0x7f63f0006bc0).stop 2021-03-09T10:53:57.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:57.742 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:53:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:53:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=20733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:53:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:53:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:57.743 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:53:57.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:57.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:53:57.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:53:57.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:53:57.744 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:57.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011340 con 0x7f63d8003400 2021-03-09T10:53:57.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:57.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b460 con 0x7f63d8009130 2021-03-09T10:53:57.745 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:53:57.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0023590 con 0x7f63e0018390 2021-03-09T10:53:57.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:53:57.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:53:57.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.738+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:57.746 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.739+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:53:57.747 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.739+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:53:57.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.739+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:53:57.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.739+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:53:57.748 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.739+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:53:57.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.739+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:53:57.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:53:57.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:53:57.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:53:57.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:53:57.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:57.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:57.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:53:57.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:53:57.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:53:57.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:53:57.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:53:57.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:57.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00230b0 con 0x7f63e0018390 2021-03-09T10:53:57.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:53:57.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:53:57.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002be90 con 0x7f63e0018390 2021-03-09T10:53:57.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:53:57.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:57.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20744 cs=0 l=1 rev1=1 rx=0x7f63f0056af0 tx=0x7f63f0039580).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:53:57.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.740+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:53:57.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.741+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:53:57.753 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:53:57.746+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61492 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:53:57.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.746+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63e0018390 2021-03-09T10:53:57.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.746+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:53:57.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.746+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:53:57.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.746+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:53:57.755 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:53:57.755 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:53:57.755 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:57.756 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:53:57.756 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:53:57.756 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:53:57.756 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:53:57.756 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:53:57.757 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:53:57.757 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:53:57.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.746+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63e0018390 2021-03-09T10:53:57.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.746+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:53:57.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.746+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:53:57.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0032c30 con 0x7f63e0018390 2021-03-09T10:53:57.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.802+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:57.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:57.802+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:58.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:58.802+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:58.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:58.802+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:53:59.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:59.802+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:53:59.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:53:59.802+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:00.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:00.800+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:00.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:00.802+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:00.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:00.802+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:00.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:00.802+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:01.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:01.803+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:01.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:01.803+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:02.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:02.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:02.749 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20744 cs=0 l=1 rev1=1 rx=0x7f63f0056af0 tx=0x7f63f0039580).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:02.750 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20744 cs=0 l=1 rev1=1 rx=0x7f63f0056af0 tx=0x7f63f0039580).stop 2021-03-09T10:54:02.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:02.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:02.751 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:02.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:02.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:54:02.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:02.752 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:02.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:02.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:02.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:02.753 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:02.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:02.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007ae70 con 0x7f63d8003400 2021-03-09T10:54:02.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:02.754 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017210 con 0x7f63d8009130 2021-03-09T10:54:02.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:02.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:54:02.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:02.755 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:02.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:02.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:02.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:02.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:02.756 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:54:02.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:02.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:02.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:02.757 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:02.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.747+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:02.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:02.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:02.758 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:02.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:02.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:02.759 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:54:02.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:54:02.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:02.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:02.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002b710 con 0x7f63d8003840 2021-03-09T10:54:02.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:02.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:02.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:54:02.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:54:02.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:02.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 secure :-1 s=READY pgs=20756 cs=0 l=1 rev1=1 rx=0x7f63f4034f70 tx=0x7f63f4062310).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:02.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:54:02.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.748+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:02.763 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:02.757+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61591 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:02.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4009920 con 0x7f63d8003840 2021-03-09T10:54:02.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.757+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.757+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.757+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:02.765 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:02.766 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:02.766 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:02.766 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:02.767 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:02.767 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:02.767 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:02.767 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:02.768 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:02.768 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:02.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.758+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4009920 con 0x7f63d8003840 2021-03-09T10:54:02.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.758+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:02.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.758+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:54:02.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.759+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022360 con 0x7f63d8003840 2021-03-09T10:54:02.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.802+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:02.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:02.802+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:03.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:03.803+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:03.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:03.803+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:04.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:04.803+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:04.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:04.803+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:05.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:05.801+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:05.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:05.803+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:05.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:05.803+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:06.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:06.803+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:06.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:06.803+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 secure :-1 s=READY pgs=20756 cs=0 l=1 rev1=1 rx=0x7f63f4034f70 tx=0x7f63f4062310).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:07.760 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 secure :-1 s=READY pgs=20756 cs=0 l=1 rev1=1 rx=0x7f63f4034f70 tx=0x7f63f4062310).stop 2021-03-09T10:54:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e001abe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e001abe0 unknown :-1 s=CLOSED pgs=20756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:07.761 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:54:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:07.762 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:07.763 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0018250 con 0x7f63d8003400 2021-03-09T10:54:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017210 con 0x7f63d8009130 2021-03-09T10:54:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63e0018390 2021-03-09T10:54:07.764 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.757+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:07.765 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:54:07.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:07.766 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:07.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:07.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:07.767 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:07.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:07.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:07.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:07.768 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.758+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:07.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:07.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:07.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:07.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:54:07.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:54:07.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:07.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:07.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f63e0018390 2021-03-09T10:54:07.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:07.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:07.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002b710 con 0x7f63e0018390 2021-03-09T10:54:07.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:54:07.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:07.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20770 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4022710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:07.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:54:07.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.759+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:07.772 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:07.766+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61685 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:07.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.767+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022a80 con 0x7f63e0018390 2021-03-09T10:54:07.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.767+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:07.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.767+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:07.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.767+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:07.774 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:07.774 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:07.775 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:07.775 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:07.775 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:07.775 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:07.776 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:07.776 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:07.776 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:07.776 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:07.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.770+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005480 con 0x7f63e0018390 2021-03-09T10:54:07.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.770+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:07.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.771+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:54:07.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.773+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022a80 con 0x7f63e0018390 2021-03-09T10:54:07.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.803+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:07.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:07.803+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:08.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:08.804+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:08.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:08.804+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:09.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:09.804+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:09.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:09.804+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:10.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:10.802+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:10.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:10.803+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:10.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:10.803+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:10.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:10.803+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:11.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:11.804+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:11.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:11.804+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:12.769 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20770 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4022710).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:12.770 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20770 cs=0 l=1 rev1=1 rx=0x7f63f4003740 tx=0x7f63f4022710).stop 2021-03-09T10:54:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:12.771 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:54:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:12.772 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:12.773 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:54:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:54:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014080 con 0x7f63d8003840 2021-03-09T10:54:12.774 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.766+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:12.775 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:54:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:12.776 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:12.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:12.777 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.767+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:12.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:12.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:12.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:12.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:12.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:12.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:54:12.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:54:12.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:12.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:12.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8003840 2021-03-09T10:54:12.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:12.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:12.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63d8003840 2021-03-09T10:54:12.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:54:12.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.768+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:12.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.769+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=20784 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003bb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:12.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.769+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:54:12.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.769+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:12.782 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:12.775+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61783 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:12.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63d8003840 2021-03-09T10:54:12.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.775+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:12.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.775+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:12.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.775+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:12.784 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:12.784 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:12.785 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:12.785 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:12.785 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:12.785 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:12.785 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:12.786 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:12.786 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:12.786 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:12.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b680 con 0x7f63d8003840 2021-03-09T10:54:12.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.775+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:12.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.775+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:54:12.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.776+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40532c0 con 0x7f63d8003840 2021-03-09T10:54:12.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.804+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:12.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:12.804+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:13.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:13.804+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:13.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:13.804+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:14.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:14.804+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:14.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:14.804+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:15.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:15.803+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:15.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:15.804+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:15.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:15.804+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:16.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:16.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:16.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:16.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:17.778 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:17.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:17.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=20784 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003bb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:17.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 secure :-1 s=READY pgs=20784 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f4003bb0).stop 2021-03-09T10:54:17.779 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:17.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:17.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0012350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:17.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0012350 unknown :-1 s=CLOSED pgs=20784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:17.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:54:17.780 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:17.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:17.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:17.781 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:17.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:17.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:17.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:17.782 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:54:17.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:17.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00096f0 con 0x7f63d8009130 2021-03-09T10:54:17.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:17.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001e3b0 con 0x7f63e0018390 2021-03-09T10:54:17.783 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:17.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:17.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.775+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:17.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.776+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:17.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.776+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:17.784 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.776+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:17.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.776+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:54:17.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.776+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:17.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.776+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:17.785 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.776+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:17.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.776+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:17.786 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.776+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:17.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:17.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:17.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:17.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:17.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:17.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:54:17.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:54:17.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:17.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:17.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63e0018390 2021-03-09T10:54:17.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:17.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:17.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:54:17.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:54:17.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:17.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:54:17.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20798 cs=0 l=1 rev1=1 rx=0x7f63f405b610 tx=0x7f63f401bf40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:17.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.777+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:17.792 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:17.785+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61884 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:17.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404cde0 con 0x7f63e0018390 2021-03-09T10:54:17.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.785+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:17.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.785+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:17.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.785+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:17.794 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:17.794 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:17.794 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:17.795 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:17.795 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:17.795 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:17.795 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:17.796 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:17.796 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:17.796 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:17.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f405b220 con 0x7f63e0018390 2021-03-09T10:54:17.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.785+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:17.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.785+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:54:17.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.786+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f401b2e0 con 0x7f63e0018390 2021-03-09T10:54:17.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:17.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:17.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:18.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:18.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:18.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:18.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:19.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:19.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:19.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:19.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:20.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:20.804+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:20.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:20.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:20.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:20.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:20.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:20.805+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:21.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:21.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:21.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:21.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:22.787 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:22.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:22.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20798 cs=0 l=1 rev1=1 rx=0x7f63f405b610 tx=0x7f63f401bf40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:22.788 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20798 cs=0 l=1 rev1=1 rx=0x7f63f405b610 tx=0x7f63f401bf40).stop 2021-03-09T10:54:22.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:22.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:22.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:22.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:22.789 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:54:22.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:22.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:22.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:22.790 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:22.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:22.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:22.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:22.791 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028f40 con 0x7f63d8003400 2021-03-09T10:54:22.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:22.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:54:22.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:22.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001c590 con 0x7f63d8003840 2021-03-09T10:54:22.792 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:22.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:22.793 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.785+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:22.794 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.786+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:22.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.786+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:54:22.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.786+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:22.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.786+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:22.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.786+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:22.795 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.786+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:22.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.786+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:22.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:22.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2021-03-09T10:54:22.796 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:22.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:22.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:22.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:54:22.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:54:22.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:22.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:22.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00166d0 con 0x7f63d8003840 2021-03-09T10:54:22.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:22.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:22.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00230b0 con 0x7f63d8003840 2021-03-09T10:54:22.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:54:22.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:22.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.787+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=20812 cs=0 l=1 rev1=1 rx=0x7f63e0079460 tx=0x7f63f00395f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:22.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.788+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:54:22.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.788+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:22.801 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:22.794+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 61982 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:22.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f004dab0 con 0x7f63d8003840 2021-03-09T10:54:22.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.795+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:22.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.795+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:22.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.795+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:22.802 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:22.803 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:22.803 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:22.803 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:22.803 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:22.804 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:22.804 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:22.804 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:22.804 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:22.804 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:22.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00393e0 con 0x7f63d8003840 2021-03-09T10:54:22.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.795+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:22.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.795+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:54:22.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.796+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0032c30 con 0x7f63d8003840 2021-03-09T10:54:22.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:22.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:22.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:23.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:23.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:23.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:24.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:24.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:24.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:24.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:25.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:25.805+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:25.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:25.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:25.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:25.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:26.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:26.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:26.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:26.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:27.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:27.797 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:27.798 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=20812 cs=0 l=1 rev1=1 rx=0x7f63e0079460 tx=0x7f63f00395f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:27.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=20812 cs=0 l=1 rev1=1 rx=0x7f63e0079460 tx=0x7f63f00395f0).stop 2021-03-09T10:54:27.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:27.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:27.799 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:27.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=CLOSED pgs=20812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:27.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:54:27.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:27.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:27.800 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:27.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:27.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:27.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:27.801 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:27.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:54:27.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:27.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028580 con 0x7f63d8009130 2021-03-09T10:54:27.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:27.802 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022d90 con 0x7f63e0018390 2021-03-09T10:54:27.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:27.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:27.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:27.803 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:27.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.795+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:27.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.796+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:27.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.796+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:54:27.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.796+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:27.804 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.796+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:27.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:27.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:27.805 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:27.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:27.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:27.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:27.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:27.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:27.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:54:27.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:54:27.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:27.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:27.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:54:27.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:27.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:27.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a350 con 0x7f63e0018390 2021-03-09T10:54:27.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:54:27.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.797+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:27.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.798+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:54:27.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.798+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:27.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.798+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20827 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc065870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:27.810 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:27.803+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62083 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:27.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.803+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63e0018390 2021-03-09T10:54:27.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.803+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:27.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.803+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:27.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.803+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:27.812 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:27.812 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:27.812 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:27.813 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:27.813 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:27.813 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:27.813 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:27.813 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:27.814 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:27.814 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:27.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:27.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:27.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.806+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63e0018390 2021-03-09T10:54:27.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.806+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:27.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.807+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:54:27.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:27.807+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:54:28.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:28.805+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:28.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:28.805+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:29.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:29.806+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:29.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:29.806+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:30.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:30.806+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:30.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:30.806+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:30.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:30.806+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:30.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:30.806+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:31.806+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:31.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:31.806+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:32.806 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:32.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:32.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20827 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc065870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:32.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20827 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc065870).stop 2021-03-09T10:54:32.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:32.807 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:32.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:32.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:32.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:54:32.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:32.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:32.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:32.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:32.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:32.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:32.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:32.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a220 con 0x7f63d8003400 2021-03-09T10:54:32.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:32.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:54:32.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:32.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027810 con 0x7f63d8003840 2021-03-09T10:54:32.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:32.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:32.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:32.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:32.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.804+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:32.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:32.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:54:32.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:32.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:32.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:32.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:32.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:32.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:32.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:32.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:32.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:32.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:32.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:54:32.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:54:32.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:32.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:32.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00241d0 con 0x7f63d8003840 2021-03-09T10:54:32.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:32.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:32.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00166d0 con 0x7f63d8003840 2021-03-09T10:54:32.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:54:32.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:32.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065700 secure :-1 s=READY pgs=20838 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:32.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:54:32.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.805+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:32.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.806+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:32.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.806+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:32.820 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:32.814+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62181 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:32.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:54:32.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.814+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:32.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.814+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:32.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.814+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:32.822 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:32.822 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:32.823 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:32.823 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:32.823 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:32.823 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:32.824 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:32.824 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:32.824 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:32.824 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:32.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:54:32.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.814+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:32.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.814+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:54:32.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:32.815+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:54:33.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:33.806+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:33.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:33.806+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:34.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:34.807+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:34.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:34.807+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:35.808 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:35.806+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:35.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:35.806+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:35.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:35.807+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:36.807+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:36.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:36.807+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.807+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:37.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.807+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:37.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:37.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:37.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065700 secure :-1 s=READY pgs=20838 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:37.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065700 secure :-1 s=READY pgs=20838 cs=0 l=1 rev1=1 rx=0x7f63f002d640 tx=0x7f63f004db20).stop 2021-03-09T10:54:37.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:37.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:37.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0065700 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:37.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0065700 unknown :-1 s=CLOSED pgs=20838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:37.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:54:37.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:37.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:37.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:37.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:37.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:37.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:37.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:37.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007be60 con 0x7f63d8003400 2021-03-09T10:54:37.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:37.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:54:37.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:37.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0028580 con 0x7f63e0018390 2021-03-09T10:54:37.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:37.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:37.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:37.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:37.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.814+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:37.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.815+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:37.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.815+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:54:37.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.815+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:37.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.815+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:37.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:37.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:37.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:37.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:37.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:37.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:37.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:37.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:37.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:54:37.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:54:37.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006b010 con 0x7f63e0018390 2021-03-09T10:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:37.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.816+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.817+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20854 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc052f70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.817+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:54:37.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.817+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:37.829 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:37.822+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62282 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:37.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc051610 con 0x7f63e0018390 2021-03-09T10:54:37.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.823+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:37.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.823+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:37.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.823+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:37.831 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:37.832 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:37.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.826+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051610 con 0x7f63e0018390 2021-03-09T10:54:37.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.826+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:37.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.826+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:54:37.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:37.826+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:54:38.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:38.807+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:38.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:38.807+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:39.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:39.807+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:39.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:39.807+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:40.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:40.807+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:40.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:40.807+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:40.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:40.807+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:40.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:40.808+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:41.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:41.808+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:41.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:41.808+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:42.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.808+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:42.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.808+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:42.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:42.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:42.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20854 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc052f70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:42.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20854 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc052f70).stop 2021-03-09T10:54:42.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:42.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:42.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:42.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20854 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:42.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:54:42.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:42.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:42.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:42.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:42.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:42.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:42.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:42.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013920 con 0x7f63d8003400 2021-03-09T10:54:42.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:42.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a220 con 0x7f63d8009130 2021-03-09T10:54:42.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:42.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027810 con 0x7f63d8003840 2021-03-09T10:54:42.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:54:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.823+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:42.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:42.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:42.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:42.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:42.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:42.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:42.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:54:42.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:54:42.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:42.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:42.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0016220 con 0x7f63d8003840 2021-03-09T10:54:42.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:42.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:42.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00241d0 con 0x7f63d8003840 2021-03-09T10:54:42.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:54:42.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.824+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:42.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.825+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:54:42.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.825+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:42.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.825+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 secure :-1 s=READY pgs=20866 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0057f50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:42.841 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:42.833+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62381 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:42.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.833+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:54:42.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.833+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:42.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.833+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:42.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.833+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:42.843 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:42.843 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:42.844 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:42.844 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:42.844 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:42.844 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:42.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.834+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63d8003840 2021-03-09T10:54:42.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.834+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:42.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.834+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:54:42.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:42.835+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0011da0 con 0x7f63d8003840 2021-03-09T10:54:43.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:43.808+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:43.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:43.808+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:44.809 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:44.808+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:44.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:44.808+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:45.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:45.808+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:45.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:45.808+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:45.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:45.809+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:46.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:46.808+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:46.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:46.808+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:47.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.808+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:47.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.808+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:47.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:47.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:47.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 secure :-1 s=READY pgs=20866 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0057f50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 secure :-1 s=READY pgs=20866 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f0057f50).stop 2021-03-09T10:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006b260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:47.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006b260 unknown :-1 s=CLOSED pgs=20866 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:47.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:54:47.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:47.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:47.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:47.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:47.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:47.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:47.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:47.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:54:47.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:47.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065fe0 con 0x7f63d8009130 2021-03-09T10:54:47.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:47.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63e0018390 2021-03-09T10:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:54:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:47.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:47.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.834+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:47.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:47.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:47.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:54:47.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:47.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:47.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0025ba0 con 0x7f63e0018390 2021-03-09T10:54:47.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:47.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:47.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63e0018390 2021-03-09T10:54:47.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:54:47.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.835+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:47.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.836+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20883 cs=0 l=1 rev1=1 rx=0x7f63f4048da0 tx=0x7f63f40051f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:47.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.836+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:54:47.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.836+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:47.848 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:47.841+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62482 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:47.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402cef0 con 0x7f63e0018390 2021-03-09T10:54:47.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.842+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:47.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.842+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:47.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.842+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:47.850 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:47.850 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:47.850 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:47.850 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:47.851 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:47.851 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:47.851 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:47.851 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:47.851 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:47.852 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:47.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.845+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f402cef0 con 0x7f63e0018390 2021-03-09T10:54:47.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.845+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:47.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.845+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:54:47.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:47.846+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40181f0 con 0x7f63e0018390 2021-03-09T10:54:48.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:48.808+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:48.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:48.808+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:49.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:49.809+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:49.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:49.809+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:50.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:50.809+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:50.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:50.809+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:50.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:50.809+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:50.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:50.810+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:51.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:51.809+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:51.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:51.809+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:52.810 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.809+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:52.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.809+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:52.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:54:52.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:52.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20883 cs=0 l=1 rev1=1 rx=0x7f63f4048da0 tx=0x7f63f40051f0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:52.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20883 cs=0 l=1 rev1=1 rx=0x7f63f4048da0 tx=0x7f63f40051f0).stop 2021-03-09T10:54:52.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:52.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:52.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:52.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:52.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:54:52.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:52.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:52.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:52.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8003400 2021-03-09T10:54:52.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:52.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a220 con 0x7f63d8009130 2021-03-09T10:54:52.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:52.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:54:52.849 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:52.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:52.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:52.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:52.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:52.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:52.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:54:52.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:52.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.842+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:52.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:52.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:52.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:52.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:52.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:52.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:54:52.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:52.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:52.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005ef90 con 0x7f63d8003840 2021-03-09T10:54:52.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:52.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:52.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0025ba0 con 0x7f63d8003840 2021-03-09T10:54:52.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:54:52.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.844+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:52.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.845+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=20901 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:52.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.845+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:54:52.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.845+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:52.858 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:52.850+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62573 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:52.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.850+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f63d8003840 2021-03-09T10:54:52.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.850+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:52.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.850+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:52.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.850+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:52.860 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:52.860 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:52.861 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:52.861 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:52.861 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:52.861 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:52.861 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:52.862 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:52.862 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:52.862 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:52.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4017e60 con 0x7f63d8003840 2021-03-09T10:54:52.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.855+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:52.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.855+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:54:52.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:52.855+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f401ab90 con 0x7f63d8003840 2021-03-09T10:54:53.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:53.808+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:53.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:53.808+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:54.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:54.809+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:54.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:54.809+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:55.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:55.809+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:55.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:55.809+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:55.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:55.811+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:54:56.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:56.810+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:56.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:56.810+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:57.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.810+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:57.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.810+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:57.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:54:57.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:54:57.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=20901 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:54:57.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 secure :-1 s=READY pgs=20901 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4009a00).stop 2021-03-09T10:54:57.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:57.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:54:57.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000fc20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:54:57.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000fc20 unknown :-1 s=CLOSED pgs=20901 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:54:57.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:54:57.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:57.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:54:57.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:57.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:54:57.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:54:57.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:54:57.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:57.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac00 con 0x7f63d8003400 2021-03-09T10:54:57.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:57.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:54:57.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:54:57.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00195c0 con 0x7f63e0018390 2021-03-09T10:54:57.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:54:57.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:54:57.859 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:57.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:54:57.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:54:57.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:54:57.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:54:57.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:54:57.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.851+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:54:57.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:54:57.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:54:57.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:54:57.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:54:57.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:57.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:57.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:54:57.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:54:57.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:54:57.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:54:57.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:54:57.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:57.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00230b0 con 0x7f63e0018390 2021-03-09T10:54:57.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:54:57.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:54:57.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006b010 con 0x7f63e0018390 2021-03-09T10:54:57.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:54:57.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.852+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:57.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.853+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:54:57.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.853+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:54:57.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.853+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20909 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc009a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:54:57.867 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:54:57.859+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62677 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:54:57.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013e20 con 0x7f63e0018390 2021-03-09T10:54:57.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.859+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:54:57.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.859+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:54:57.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.859+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:54:57.870 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:54:57.870 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:54:57.870 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:57.871 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:54:57.871 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:54:57.871 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:54:57.872 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:54:57.872 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:54:57.873 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:54:57.873 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:54:57.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:54:57.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.859+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:54:57.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.859+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:54:57.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:57.860+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:54:58.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:58.810+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:58.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:58.810+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:54:59.811 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:59.810+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:54:59.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:54:59.810+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:00.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:00.810+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:00.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:00.810+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:00.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:00.810+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:00.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:00.812+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:01.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:01.810+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:01.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:01.810+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:02.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.810+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:02.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.810+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:02.861 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:02.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:02.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20909 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc009a00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:02.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20909 cs=0 l=1 rev1=1 rx=0x7f63fc071410 tx=0x7f63fc009a00).stop 2021-03-09T10:55:02.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:02.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:02.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:02.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:02.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:55:02.864 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:02.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:02.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:02.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:02.865 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e000 con 0x7f63d8003400 2021-03-09T10:55:02.866 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:02.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0006c60 con 0x7f63d8009130 2021-03-09T10:55:02.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:02.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:55:02.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:02.867 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:02.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:02.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:02.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.859+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:02.868 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.860+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:02.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.860+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:55:02.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.860+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:02.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.860+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:02.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.860+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:02.869 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.860+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:02.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.860+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:02.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:02.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:02.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:02.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:02.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:02.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:55:02.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:55:02.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:02.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:02.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63d8003840 2021-03-09T10:55:02.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:02.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:02.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005ef90 con 0x7f63d8003840 2021-03-09T10:55:02.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:55:02.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.861+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.862+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=20924 cs=0 l=1 rev1=1 rx=0x7f63f4003b80 tx=0x7f63f405b290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.862+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:55:02.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.862+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:02.875 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:02.867+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62775 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:02.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4034d40 con 0x7f63d8003840 2021-03-09T10:55:02.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.868+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:02.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.868+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:02.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.868+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:02.876 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:02.877 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:02.877 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:02.877 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:02.877 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:02.878 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:02.878 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:02.878 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:02.878 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:02.878 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:02.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.872+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4046e40 con 0x7f63d8003840 2021-03-09T10:55:02.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.872+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:02.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.872+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:55:02.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:02.873+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f405cdc0 con 0x7f63d8003840 2021-03-09T10:55:03.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:03.811+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:03.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:03.811+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:04.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:04.811+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:04.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:04.811+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:05.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:05.811+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:05.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:05.811+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:05.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:05.813+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:06.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:06.811+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:06.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:06.811+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:07.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.811+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:07.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.811+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:07.870 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:07.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:07.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=20924 cs=0 l=1 rev1=1 rx=0x7f63f4003b80 tx=0x7f63f405b290).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:07.871 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 secure :-1 s=READY pgs=20924 cs=0 l=1 rev1=1 rx=0x7f63f4003b80 tx=0x7f63f405b290).stop 2021-03-09T10:55:07.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:07.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:07.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0023390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:07.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0023390 unknown :-1 s=CLOSED pgs=20924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:07.872 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:55:07.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:07.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:07.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:07.873 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:07.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:07.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:07.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:07.874 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00195c0 con 0x7f63d8003400 2021-03-09T10:55:07.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:07.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012550 con 0x7f63d8009130 2021-03-09T10:55:07.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:07.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63e0018390 2021-03-09T10:55:07.875 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:07.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:07.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:07.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:07.876 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.868+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:07.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.869+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:07.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.869+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:55:07.878 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.869+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:07.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.869+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:07.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.869+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:07.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.869+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:07.879 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.869+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:07.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:07.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:07.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:07.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:07.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:07.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:55:07.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:55:07.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:07.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:07.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00120e0 con 0x7f63e0018390 2021-03-09T10:55:07.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:07.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:07.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0016220 con 0x7f63e0018390 2021-03-09T10:55:07.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:55:07.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.870+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:07.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.871+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20936 cs=0 l=1 rev1=1 rx=0x7f63f00906b0 tx=0x7f63f0013c10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:07.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.871+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:55:07.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.871+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:07.884 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:07.876+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62876 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:07.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.878+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f007c520 con 0x7f63e0018390 2021-03-09T10:55:07.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.878+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:07.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.878+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:07.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.878+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:07.886 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:07.886 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:07.886 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:07.887 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:07.887 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:07.887 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:07.887 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:07.887 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:07.888 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:07.888 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:07.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.880+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f007c520 con 0x7f63e0018390 2021-03-09T10:55:07.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.881+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:07.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.881+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:55:07.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:07.883+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63e0018390 2021-03-09T10:55:08.812 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:08.811+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:08.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:08.811+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:09.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:09.811+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:09.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:09.811+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:10.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:10.811+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:10.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:10.811+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:10.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:10.811+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:10.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:10.814+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:11.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:11.812+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:11.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:11.812+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:12.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.812+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:12.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.812+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:12.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:12.880 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:12.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20936 cs=0 l=1 rev1=1 rx=0x7f63f00906b0 tx=0x7f63f0013c10).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:12.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20936 cs=0 l=1 rev1=1 rx=0x7f63f00906b0 tx=0x7f63f0013c10).stop 2021-03-09T10:55:12.881 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:12.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:12.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:12.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:12.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:55:12.882 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:12.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:12.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:12.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:12.883 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:12.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:12.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:12.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e220 con 0x7f63d8003400 2021-03-09T10:55:12.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:12.884 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:55:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00195c0 con 0x7f63d8003840 2021-03-09T10:55:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:12.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:12.886 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:55:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:12.887 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.877+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:12.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:12.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:12.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:12.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:12.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:55:12.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:55:12.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:12.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:12.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8003840 2021-03-09T10:55:12.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:12.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:12.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00230b0 con 0x7f63d8003840 2021-03-09T10:55:12.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:55:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 secure :-1 s=READY pgs=20950 cs=0 l=1 rev1=1 rx=0x7f63fc0620c0 tx=0x7f63fc065f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:55:12.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.878+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:12.893 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:12.885+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 62974 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc012240 con 0x7f63d8003840 2021-03-09T10:55:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.885+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:12.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.885+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:12.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.885+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:12.895 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:12.895 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:12.895 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:12.895 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:12.896 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:12.896 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:12.896 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:12.896 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:12.897 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:12.897 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:12.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.885+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:55:12.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.885+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:12.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.885+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:55:12.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:12.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc078230 con 0x7f63d8003840 2021-03-09T10:55:13.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:13.811+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:13.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:13.811+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:14.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:14.812+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:14.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:14.812+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:15.813 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:15.812+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:15.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:15.812+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:15.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:15.815+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:16.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:16.812+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:16.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:16.812+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:17.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.812+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:17.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.812+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:17.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:17.888 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:17.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 secure :-1 s=READY pgs=20950 cs=0 l=1 rev1=1 rx=0x7f63fc0620c0 tx=0x7f63fc065f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:17.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 secure :-1 s=READY pgs=20950 cs=0 l=1 rev1=1 rx=0x7f63fc0620c0 tx=0x7f63fc065f90).stop 2021-03-09T10:55:17.889 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:17.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:17.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b010 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:17.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b010 unknown :-1 s=CLOSED pgs=20950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:17.890 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:55:17.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:17.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:17.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:17.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:17.891 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:17.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:17.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:17.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f610 con 0x7f63d8003400 2021-03-09T10:55:17.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:17.892 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63d8009130 2021-03-09T10:55:17.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:17.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00278d0 con 0x7f63e0018390 2021-03-09T10:55:17.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:17.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:17.893 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:17.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:17.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:17.894 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:17.895 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:55:17.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:17.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.886+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:17.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.887+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:17.896 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.887+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:17.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.887+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:17.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:17.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:17.897 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:17.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:17.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:17.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:55:17.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:55:17.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:17.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:17.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ed60 con 0x7f63e0018390 2021-03-09T10:55:17.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:17.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:17.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63e0018390 2021-03-09T10:55:17.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:55:17.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.888+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:17.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.889+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:55:17.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.889+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:17.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.889+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20967 cs=0 l=1 rev1=1 rx=0x7f63e0012b80 tx=0x7f63fc066b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:17.902 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:17.894+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63075 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:17.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc03f280 con 0x7f63e0018390 2021-03-09T10:55:17.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.895+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:17.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.895+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:17.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.895+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:17.904 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:17.904 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:17.905 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:17.905 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:17.905 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:17.906 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:17.906 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:17.906 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:17.906 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:17.907 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:17.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03f280 con 0x7f63e0018390 2021-03-09T10:55:17.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.900+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:17.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.900+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:55:17.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:17.900+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63e0018390 2021-03-09T10:55:18.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:18.813+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:18.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:18.813+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:19.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:19.813+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:19.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:19.813+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:20.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:20.813+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:20.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:20.813+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:20.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:20.813+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:20.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:20.816+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:21.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:21.813+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:21.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:21.813+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:22.814 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.812+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:22.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.812+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:22.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:22.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:22.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20967 cs=0 l=1 rev1=1 rx=0x7f63e0012b80 tx=0x7f63fc066b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:22.898 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=20967 cs=0 l=1 rev1=1 rx=0x7f63e0012b80 tx=0x7f63fc066b30).stop 2021-03-09T10:55:22.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:22.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:22.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:22.899 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=20967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:22.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:55:22.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:22.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:22.900 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:22.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:22.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:22.901 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:22.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:22.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e2c0 con 0x7f63d8003400 2021-03-09T10:55:22.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:22.902 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8009130 2021-03-09T10:55:22.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:22.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029140 con 0x7f63d8003840 2021-03-09T10:55:22.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:22.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:22.903 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:22.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:22.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:22.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:22.904 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:55:22.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:22.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.895+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:22.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.896+0000 7f6406553700 0 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-03-09T10:55:22.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:22.905 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:22.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.896+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:22.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.896+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2021-03-09T10:55:22.906 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.896+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).write auth request more write failed r=-32 ((32) Broken pipe) 2021-03-09T10:55:22.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:22.896+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:22.907 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:22.903+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63173 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:23.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.096+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:23.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.096+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T10:55:23.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.096+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:23.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.096+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2021-03-09T10:55:23.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:23.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:55:23.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:23.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:23.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:23.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:23.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:23.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:23.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:23.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:23.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:23.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:23.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:55:23.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:55:23.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:23.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:23.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005f120 con 0x7f63d8003840 2021-03-09T10:55:23.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:23.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:23.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ed60 con 0x7f63d8003840 2021-03-09T10:55:23.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:55:23.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:23.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:55:23.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:23.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ae70 secure :-1 s=READY pgs=20990 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc066e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:23.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04d5f0 con 0x7f63d8003840 2021-03-09T10:55:23.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:23.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:23.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:23.107 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:23.108 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:23.108 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:23.108 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:23.108 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:23.109 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:23.109 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:23.109 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:23.109 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:23.110 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:23.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03f280 con 0x7f63d8003840 2021-03-09T10:55:23.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:23.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.097+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:55:23.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.098+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc0111a0 con 0x7f63d8003840 2021-03-09T10:55:23.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.813+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:23.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:23.813+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:24.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:24.813+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:24.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:24.813+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:25.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:25.814+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:25.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:25.814+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:25.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:25.817+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:26.813+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:26.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:26.813+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:27.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.814+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:27.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.814+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:27.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.904+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:27.907 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.904+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:27.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.904+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ae70 secure :-1 s=READY pgs=20990 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc066e40).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:27.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.904+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ae70 secure :-1 s=READY pgs=20990 cs=0 l=1 rev1=1 rx=0x7f63fc009ab0 tx=0x7f63fc066e40).stop 2021-03-09T10:55:27.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:27.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:27.908 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007ae70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:27.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007ae70 unknown :-1 s=CLOSED pgs=20990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:27.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:55:27.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:27.909 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:27.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:27.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:27.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:27.910 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:27.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:27.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:55:27.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:27.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:55:27.911 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:27.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f610 con 0x7f63e0018390 2021-03-09T10:55:27.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:27.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:27.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:27.912 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:27.913 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.905+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:27.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.906+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:27.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.906+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:55:27.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.906+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:27.915 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.906+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:27.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.907+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:27.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.907+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:27.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.907+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:27.916 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:27.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:27.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:27.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:27.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:27.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:55:27.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:55:27.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:27.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:27.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ed60 con 0x7f63e0018390 2021-03-09T10:55:27.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:27.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:27.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00120e0 con 0x7f63e0018390 2021-03-09T10:55:27.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:55:27.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:27.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21002 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f0009510).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:27.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:55:27.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.908+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:27.920 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:27.913+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63274 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:27.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0055360 con 0x7f63e0018390 2021-03-09T10:55:27.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.914+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:27.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.914+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:27.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.914+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:27.922 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:27.923 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:27.923 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:27.923 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:27.923 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:27.923 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:27.924 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:27.924 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:27.924 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:27.925 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:27.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.917+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0055360 con 0x7f63e0018390 2021-03-09T10:55:27.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.917+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:27.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.917+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:55:27.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:27.918+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:55:28.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:28.814+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:28.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:28.814+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:29.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:29.814+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:29.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:29.814+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:30.815 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:30.814+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:30.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:30.814+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:30.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:30.814+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:30.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:30.818+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:31.814+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:31.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:31.814+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:32.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.814+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:32.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.814+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:32.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:32.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:32.917 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21002 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f0009510).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:32.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21002 cs=0 l=1 rev1=1 rx=0x7f63f0013c80 tx=0x7f63f0009510).stop 2021-03-09T10:55:32.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:32.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:32.918 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:32.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:32.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:55:32.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:32.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:32.919 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:32.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:32.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:32.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:32.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:32.920 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a4b0 con 0x7f63d8003400 2021-03-09T10:55:32.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:32.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079550 con 0x7f63d8009130 2021-03-09T10:55:32.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:32.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0029140 con 0x7f63d8003840 2021-03-09T10:55:32.921 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:32.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:32.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.914+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:32.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.915+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:32.922 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.915+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:32.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.915+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:32.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.915+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:55:32.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.915+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:32.923 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.915+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:32.924 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:32.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:32.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:32.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:32.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:32.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:32.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:32.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:32.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:55:32.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:55:32.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:32.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:32.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005ef30 con 0x7f63d8003840 2021-03-09T10:55:32.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:32.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:32.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005f120 con 0x7f63d8003840 2021-03-09T10:55:32.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:55:32.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.916+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:32.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.917+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b050 secure :-1 s=READY pgs=21009 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc02df80).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:32.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.917+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:55:32.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.917+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:32.929 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:32.922+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63372 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:32.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.923+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc013c60 con 0x7f63d8003840 2021-03-09T10:55:32.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.923+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:32.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.923+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:32.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.923+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:32.931 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:32.931 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:32.931 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:32.932 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:32.932 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:32.932 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:32.932 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:32.932 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:32.933 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:32.933 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:32.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.926+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0231f0 con 0x7f63d8003840 2021-03-09T10:55:32.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.926+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:32.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.926+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:55:32.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:32.928+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:55:33.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:33.814+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:33.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:33.814+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:34.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:34.815+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:34.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:34.815+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:35.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:35.815+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:35.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:35.815+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:35.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:35.819+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:36.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:36.815+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:36.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:36.815+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:37.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.815+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:37.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.815+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:37.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:37.925 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:37.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b050 secure :-1 s=READY pgs=21009 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc02df80).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:37.926 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b050 secure :-1 s=READY pgs=21009 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc02df80).stop 2021-03-09T10:55:37.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:37.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:37.927 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007b050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:37.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007b050 unknown :-1 s=CLOSED pgs=21009 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:37.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:55:37.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:37.928 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:37.929 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:37.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:37.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:37.930 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:37.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:37.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001e90 con 0x7f63d8003400 2021-03-09T10:55:37.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:37.931 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001ba00 con 0x7f63d8009130 2021-03-09T10:55:37.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:37.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:55:37.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:37.932 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:37.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:37.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.923+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:37.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.924+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:37.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.924+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:37.933 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.925+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:55:37.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.925+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:37.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.925+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:37.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.925+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:37.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.925+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:37.934 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.925+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:37.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:37.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:37.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:37.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:37.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:37.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:55:37.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:55:37.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:37.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:37.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002a380 con 0x7f63e0018390 2021-03-09T10:55:37.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:37.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:37.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:55:37.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:55:37.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.926+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:37.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.927+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:55:37.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.927+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:37.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.927+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21026 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40623b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:37.941 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:37.932+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63471 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:37.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.932+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022570 con 0x7f63e0018390 2021-03-09T10:55:37.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.932+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:37.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.932+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:37.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.932+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:37.943 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:37.943 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:37.943 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:37.944 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:37.945 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:37.945 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:37.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022080 con 0x7f63e0018390 2021-03-09T10:55:37.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.937+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:37.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.937+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:55:37.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:37.937+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0018390 2021-03-09T10:55:38.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:38.815+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:38.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:38.815+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:39.816 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:39.815+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:39.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:39.815+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:40.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:40.815+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:40.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:40.815+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:40.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:40.816+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:40.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:40.820+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:41.816+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:41.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:41.816+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:42.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.816+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:42.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.816+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:42.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:42.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:42.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21026 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40623b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:42.935 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21026 cs=0 l=1 rev1=1 rx=0x7f63f4000ea0 tx=0x7f63f40623b0).stop 2021-03-09T10:55:42.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:42.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:42.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:42.936 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:42.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:55:42.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:42.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:42.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:42.937 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:42.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:42.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:42.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:42.938 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e000d890 con 0x7f63d8003400 2021-03-09T10:55:42.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:42.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:55:42.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:42.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a4b0 con 0x7f63d8003840 2021-03-09T10:55:42.939 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:42.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:42.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:42.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:42.940 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:42.941 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.933+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:42.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.934+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:55:42.942 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.934+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:42.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.934+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:42.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:42.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:42.943 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.934+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:42.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:42.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:42.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:42.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:42.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:42.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:55:42.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:55:42.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:42.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:42.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00298e0 con 0x7f63d8003840 2021-03-09T10:55:42.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:42.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:42.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002a380 con 0x7f63d8003840 2021-03-09T10:55:42.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:55:42.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:42.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 secure :-1 s=READY pgs=21034 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f4072e70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:42.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:55:42.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.935+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:42.948 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:42.941+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63571 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:42.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4016200 con 0x7f63d8003840 2021-03-09T10:55:42.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.942+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:42.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.942+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:42.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.942+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:42.950 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:42.950 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:42.950 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:42.951 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:42.952 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:42.952 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:42.952 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:42.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.943+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4016200 con 0x7f63d8003840 2021-03-09T10:55:42.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.943+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:42.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.943+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:55:42.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:42.944+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022a80 con 0x7f63d8003840 2021-03-09T10:55:43.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:43.816+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:43.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:43.816+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:44.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:44.816+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:44.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:44.816+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:45.817 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:45.816+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:45.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:45.816+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:45.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:45.821+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:46.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:46.816+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:46.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:46.816+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:47.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.816+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:47.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.816+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:47.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.941+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:55:47.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.941+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:47.944 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.941+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 secure :-1 s=READY pgs=21034 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f4072e70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:47.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.941+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 secure :-1 s=READY pgs=21034 cs=0 l=1 rev1=1 rx=0x7f63f4003930 tx=0x7f63f4072e70).stop 2021-03-09T10:55:47.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:47.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:47.945 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00114d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:47.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00114d0 unknown :-1 s=CLOSED pgs=21034 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:47.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:55:47.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:47.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:47.946 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:47.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:47.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:47.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:47.947 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:47.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079550 con 0x7f63d8003400 2021-03-09T10:55:47.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:47.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:55:47.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:47.948 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002d420 con 0x7f63e0018390 2021-03-09T10:55:47.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:47.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:47.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:47.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:47.949 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.942+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:47.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.943+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:47.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.943+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:55:47.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.943+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:47.950 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.943+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:47.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.943+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:47.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.943+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:47.951 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.943+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:47.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.943+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:47.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:47.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:47.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:47.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:47.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:55:47.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:55:47.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:47.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:47.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00166d0 con 0x7f63e0018390 2021-03-09T10:55:47.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:47.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:47.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ed60 con 0x7f63e0018390 2021-03-09T10:55:47.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:55:47.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:47.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:55:47.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:47.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.944+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21046 cs=0 l=1 rev1=1 rx=0x7f63f00668e0 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:47.956 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:47.950+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63670 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:47.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.953+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00664f0 con 0x7f63e0018390 2021-03-09T10:55:47.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.953+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:47.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.953+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:47.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.953+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:47.958 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:47.958 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:47.959 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:47.959 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:47.959 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:47.959 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:47.959 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:47.960 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:47.960 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:47.960 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:47.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.954+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0066310 con 0x7f63e0018390 2021-03-09T10:55:47.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.954+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:47.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.954+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:55:47.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:47.955+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d30 con 0x7f63e0018390 2021-03-09T10:55:48.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:48.817+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:48.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:48.817+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:49.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:49.817+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:49.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:49.817+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:50.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:50.817+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:50.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:50.817+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:50.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:50.817+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:50.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:50.822+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:51.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:51.817+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:51.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:51.817+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:52.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.817+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:52.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.817+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:52.952 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21046 cs=0 l=1 rev1=1 rx=0x7f63f00668e0 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21046 cs=0 l=1 rev1=1 rx=0x7f63f00668e0 tx=0x7f63f0000bc0).stop 2021-03-09T10:55:52.953 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21046 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:52.954 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:55:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:52.955 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.950+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:52.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:52.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:52.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:52.956 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f610 con 0x7f63d8003400 2021-03-09T10:55:52.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:52.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:55:52.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:52.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f960 con 0x7f63d8003840 2021-03-09T10:55:52.957 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:52.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:52.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:52.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:52.958 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.951+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:52.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.952+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:52.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.952+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:55:52.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.952+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:52.960 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.952+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:52.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.952+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:52.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.952+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:52.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.952+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:52.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:52.961 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:52.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:52.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:52.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:52.962 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:55:52.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:55:52.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:52.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:52.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067990 con 0x7f63d8003840 2021-03-09T10:55:52.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00166d0 con 0x7f63d8003840 2021-03-09T10:55:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:55:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.953+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:52.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.954+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c750 secure :-1 s=READY pgs=21065 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007c350).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:52.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.954+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:55:52.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.954+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:52.966 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:52.960+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63768 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:52.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00314d0 con 0x7f63d8003840 2021-03-09T10:55:52.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.960+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:52.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.960+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:52.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.960+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:52.967 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:52.967 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:52.967 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:52.968 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:52.968 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:52.968 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:52.968 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:52.968 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:52.969 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:52.969 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:52.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.966+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00312f0 con 0x7f63d8003840 2021-03-09T10:55:52.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.966+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:52.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.966+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:55:52.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:52.967+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0032c30 con 0x7f63d8003840 2021-03-09T10:55:53.818 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:53.817+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:53.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:53.817+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:54.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:54.817+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:54.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:54.817+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:55.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:55.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:55.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:55.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:55.823+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:55:56.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:56.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:56.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:56.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:57.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:57.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:57.963 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:55:57.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:55:57.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c750 secure :-1 s=READY pgs=21065 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007c350).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:55:57.964 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c750 secure :-1 s=READY pgs=21065 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f007c350).stop 2021-03-09T10:55:57.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:57.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:55:57.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e000c750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:55:57.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e000c750 unknown :-1 s=CLOSED pgs=21065 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:55:57.965 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:55:57.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:57.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:55:57.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e000d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:57.966 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:55:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:55:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:55:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079550 con 0x7f63d8003400 2021-03-09T10:55:57.967 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:55:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:55:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a730 con 0x7f63e0018390 2021-03-09T10:55:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.960+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:55:57.968 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:55:57.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:57.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e000d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:55:57.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:55:57.969 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:55:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:55:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:55:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:55:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:55:57.970 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:55:57.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.961+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:55:57.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:55:57.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:57.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:57.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:55:57.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:55:57.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:55:57.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:55:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:55:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00696d0 con 0x7f63e0018390 2021-03-09T10:55:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:55:57.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:55:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005ef30 con 0x7f63e0018390 2021-03-09T10:55:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:55:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:55:57.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:55:57.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.962+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e000d4c0 secure :-1 s=READY pgs=21074 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc065f90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:55:57.975 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:55:57.969+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63869 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:55:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63e0018390 2021-03-09T10:55:57.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.969+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:55:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.969+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:55:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.969+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:55:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:55:57.977 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:55:57.978 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:57.978 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:55:57.978 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:55:57.978 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:55:57.978 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:55:57.979 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:55:57.979 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:55:57.979 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:55:57.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc03dd70 con 0x7f63e0018390 2021-03-09T10:55:57.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.969+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:55:57.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.969+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:55:57.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:57.970+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:55:58.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:58.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:58.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:58.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:55:59.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:59.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:55:59.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:55:59.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:00.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:00.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:00.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:00.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:00.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:00.818+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:00.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:00.824+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:01.819 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:01.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:01.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:01.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:02.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:02.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:02.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e000d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:02.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e000d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:02.971 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e000d4c0 secure :-1 s=READY pgs=21074 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc065f90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:02.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e000d4c0 secure :-1 s=READY pgs=21074 cs=0 l=1 rev1=1 rx=0x7f63e0043660 tx=0x7f63fc065f90).stop 2021-03-09T10:56:02.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:02.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:02.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e000d4c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:02.972 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e000d4c0 unknown :-1 s=CLOSED pgs=21074 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:02.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:56:02.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:02.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:02.973 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:02.974 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:56:02.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:02.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068f90 con 0x7f63d8009130 2021-03-09T10:56:02.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:02.975 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001a730 con 0x7f63d8003840 2021-03-09T10:56:02.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.969+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:02.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.970+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:02.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.970+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:02.976 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.970+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:02.977 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.970+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:02.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.970+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:02.978 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.971+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:56:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.971+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.971+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:02.979 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:02.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:02.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:56:02.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:56:02.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:02.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:02.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002ed60 con 0x7f63d8003840 2021-03-09T10:56:02.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:02.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:02.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00696d0 con 0x7f63d8003840 2021-03-09T10:56:02.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:56:02.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.973+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:02.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.974+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:56:02.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.974+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:02.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.974+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=21097 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:02.984 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:02.978+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 63969 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:02.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.979+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c400 con 0x7f63d8003840 2021-03-09T10:56:02.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.979+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:02.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.979+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:02.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.979+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:02.986 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:02.986 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:02.986 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:02.986 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:02.987 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:02.987 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:02.987 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:02.987 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:02.987 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:02.988 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:02.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.982+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02c220 con 0x7f63d8003840 2021-03-09T10:56:02.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.982+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:02.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.982+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:56:02.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:02.984+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065450 con 0x7f63d8003840 2021-03-09T10:56:03.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:03.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:03.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:03.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:04.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:04.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:04.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:04.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:05.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:05.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:05.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:05.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:05.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:05.825+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:06.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:06.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:06.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:06.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:07.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:07.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.978+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:56:07.980 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.978+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:07.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.978+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=21097 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:07.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.978+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 secure :-1 s=READY pgs=21097 cs=0 l=1 rev1=1 rx=0x7f63fc011600 tx=0x7f63fc009990).stop 2021-03-09T10:56:07.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.978+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:07.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.978+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:07.981 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.978+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0014ae0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.978+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0014ae0 unknown :-1 s=CLOSED pgs=21097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.978+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:56:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:07.982 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:07.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:07.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:07.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:07.983 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:07.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:07.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e2e0 con 0x7f63d8003400 2021-03-09T10:56:07.984 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021ef0 con 0x7f63d8009130 2021-03-09T10:56:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079550 con 0x7f63e0018390 2021-03-09T10:56:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:07.985 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:07.986 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:07.987 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:56:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:07.988 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:07.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:07.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.979+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:07.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:07.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:07.989 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:07.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:07.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:07.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:56:07.990 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:56:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63e0018390 2021-03-09T10:56:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:07.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:07.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00298e0 con 0x7f63e0018390 2021-03-09T10:56:07.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:56:07.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.980+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:07.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.981+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:56:07.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.981+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:07.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.981+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21103 cs=0 l=1 rev1=1 rx=0x7f63f4072e40 tx=0x7f63f4009920).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:07.993 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:07.987+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64068 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:07.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4022430 con 0x7f63e0018390 2021-03-09T10:56:07.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.987+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:07.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.987+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:07.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.987+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:07.995 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:07.995 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:07.995 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:07.995 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:07.996 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:07.996 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:07.996 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:07.996 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:07.996 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:07.997 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:07.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.988+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4022430 con 0x7f63e0018390 2021-03-09T10:56:07.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.988+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:07.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.988+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:56:07.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:07.989+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63e0018390 2021-03-09T10:56:08.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:08.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:08.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:08.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:09.820 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:09.818+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:09.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:09.818+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:10.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:10.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:10.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:10.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:10.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:10.819+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:10.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:10.825+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:11.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:11.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:11.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:12.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:12.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:12.991 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.986+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:12.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.986+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:12.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.986+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21103 cs=0 l=1 rev1=1 rx=0x7f63f4072e40 tx=0x7f63f4009920).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:12.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.986+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21103 cs=0 l=1 rev1=1 rx=0x7f63f4072e40 tx=0x7f63f4009920).stop 2021-03-09T10:56:12.992 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:12.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:12.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:12.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:12.993 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:56:12.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:12.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:12.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:12.994 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:12.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:12.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:12.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:12.995 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:56:12.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:12.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b500 con 0x7f63d8009130 2021-03-09T10:56:12.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:12.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011db0 con 0x7f63d8003840 2021-03-09T10:56:12.996 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:12.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:12.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:12.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:12.997 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:12.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:12.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:56:12.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:12.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.987+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:12.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:12.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:12.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:12.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:13.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:13.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:13.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:13.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:13.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:56:13.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:56:13.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:13.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:13.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:56:13.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:13.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:13.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002ed60 con 0x7f63d8003840 2021-03-09T10:56:13.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:56:13.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.988+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:13.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.989+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:56:13.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.989+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:13.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.989+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027bf0 secure :-1 s=READY pgs=21116 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc03ff20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:13.004 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:12.995+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64168 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:13.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c220 con 0x7f63d8003840 2021-03-09T10:56:13.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.995+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:13.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.995+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:13.005 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.995+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:13.006 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:13.006 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:13.006 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:13.006 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:13.007 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:13.007 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:13.007 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:13.007 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:13.008 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:13.008 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:13.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065790 con 0x7f63d8003840 2021-03-09T10:56:13.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.995+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:13.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.995+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:56:13.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:12.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63d8003840 2021-03-09T10:56:13.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:13.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:13.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:13.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:14.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:14.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:14.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:14.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:15.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:15.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:15.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:15.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:15.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:15.826+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:16.821 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:16.819+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:16.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:16.819+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:17.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.820+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:17.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.820+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:17.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:17.998 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027bf0 secure :-1 s=READY pgs=21116 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc03ff20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027bf0 secure :-1 s=READY pgs=21116 cs=0 l=1 rev1=1 rx=0x7f63fc047c60 tx=0x7f63fc03ff20).stop 2021-03-09T10:56:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:17.999 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0027bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:18.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0027bf0 unknown :-1 s=CLOSED pgs=21116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:18.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:56:18.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:18.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:18.000 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:18.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:18.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.995+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:18.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:18.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:18.001 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63d8003400 2021-03-09T10:56:18.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:18.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0021ef0 con 0x7f63d8009130 2021-03-09T10:56:18.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:18.002 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f140 con 0x7f63e0018390 2021-03-09T10:56:18.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:18.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:18.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:18.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:18.003 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:18.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:18.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:56:18.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:18.004 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:18.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:18.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:18.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.996+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:18.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:18.006 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:18.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:18.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:18.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:18.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:56:18.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:56:18.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:18.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:18.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63e0018390 2021-03-09T10:56:18.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:18.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:18.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067990 con 0x7f63e0018390 2021-03-09T10:56:18.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:56:18.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.997+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:18.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.998+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21128 cs=0 l=1 rev1=1 rx=0x7f63e0003380 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:18.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.998+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:56:18.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:17.998+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:18.011 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:18.004+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64268 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:18.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63e0018390 2021-03-09T10:56:18.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.004+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:18.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.004+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:18.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.004+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:18.013 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:18.013 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:18.013 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:18.013 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:18.014 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:18.014 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:18.014 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:18.014 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:18.014 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:18.015 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:18.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0009510 con 0x7f63e0018390 2021-03-09T10:56:18.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.004+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:18.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.004+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:56:18.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.006+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63e0018390 2021-03-09T10:56:18.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.820+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:18.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:18.820+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:19.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:19.820+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:19.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:19.820+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:20.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:20.820+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:20.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:20.820+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:20.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:20.820+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:20.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:20.827+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:21.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:21.820+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:21.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:21.820+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:22.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:22.820+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:22.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:22.820+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:23.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.004+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:23.007 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.004+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:23.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.004+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21128 cs=0 l=1 rev1=1 rx=0x7f63e0003380 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:23.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.004+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21128 cs=0 l=1 rev1=1 rx=0x7f63e0003380 tx=0x7f63f004db20).stop 2021-03-09T10:56:23.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.004+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:23.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.004+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:23.008 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.004+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:23.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.004+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:23.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:56:23.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0028300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:23.009 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:23.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:23.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:23.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:23.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:23.010 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:23.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079550 con 0x7f63d8003400 2021-03-09T10:56:23.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:23.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0012790 con 0x7f63d8009130 2021-03-09T10:56:23.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:23.011 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:56:23.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:23.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:23.012 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:23.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:23.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:23.014 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0028300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:23.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:56:23.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:23.015 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.005+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:23.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.006+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:23.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.006+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:23.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.006+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:23.016 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:23.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:23.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:23.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:23.017 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:23.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:56:23.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:56:23.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:23.018 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:23.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017e80 con 0x7f63d8003840 2021-03-09T10:56:23.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:23.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:23.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:56:23.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:56:23.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.007+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:23.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.008+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0028300 secure :-1 s=READY pgs=21147 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f400eed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:23.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.008+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:56:23.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.008+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:23.022 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:23.013+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64367 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:23.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.014+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4072ee0 con 0x7f63d8003840 2021-03-09T10:56:23.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.014+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:23.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.014+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:23.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.014+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:23.023 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:23.024 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:23.024 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:23.024 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:23.024 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:23.024 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:23.025 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:23.025 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:23.025 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:23.025 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:23.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4072ee0 con 0x7f63d8003840 2021-03-09T10:56:23.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.019+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:23.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.019+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:56:23.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.019+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63d8003840 2021-03-09T10:56:23.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.820+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:23.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:23.820+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:24.822 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:24.820+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:24.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:24.820+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:25.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:25.820+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:25.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:25.820+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:25.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:25.828+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:26.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:26.821+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:26.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:26.821+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:27.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:27.821+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:27.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:27.821+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:28.019 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.013+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0028300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:28.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.013+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0028300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:28.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.013+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0028300 secure :-1 s=READY pgs=21147 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f400eed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:28.020 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0028300 secure :-1 s=READY pgs=21147 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f400eed0).stop 2021-03-09T10:56:28.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:28.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:28.021 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e0028300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:28.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e0028300 unknown :-1 s=CLOSED pgs=21147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:28.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:56:28.022 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:28.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:28.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:28.023 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:28.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:28.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:28.024 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:28.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e420 con 0x7f63d8003400 2021-03-09T10:56:28.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:28.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068f90 con 0x7f63d8009130 2021-03-09T10:56:28.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:28.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0066510 con 0x7f63e0018390 2021-03-09T10:56:28.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:28.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:28.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:28.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:28.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:28.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:28.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:56:28.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:28.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.014+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:28.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:28.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:28.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:28.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:28.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:28.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:28.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:28.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:28.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:56:28.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:56:28.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:28.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:28.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0067990 con 0x7f63e0018390 2021-03-09T10:56:28.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:28.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:28.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:56:28.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:56:28.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.015+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:28.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.016+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21156 cs=0 l=1 rev1=1 rx=0x7f63fc047a90 tx=0x7f63fc04a7b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:28.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.016+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:56:28.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.016+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:28.036 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:28.022+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64463 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:28.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.022+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc029e20 con 0x7f63e0018390 2021-03-09T10:56:28.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.022+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:28.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.022+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:28.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.022+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:28.037 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:28.038 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:28.038 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:28.038 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:28.038 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:28.038 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:28.039 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:28.039 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:28.039 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:28.039 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:28.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.022+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:56:28.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.022+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:28.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.022+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:56:28.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.023+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:56:28.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.821+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:28.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:28.821+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:29.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:29.821+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:29.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:29.821+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:30.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:30.821+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:30.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:30.821+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:30.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:30.821+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:30.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:30.829+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:31.823 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:31.821+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:31.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:31.821+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:32.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:32.822+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:32.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:32.822+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:33.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:33.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:33.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21156 cs=0 l=1 rev1=1 rx=0x7f63fc047a90 tx=0x7f63fc04a7b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:33.025 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21156 cs=0 l=1 rev1=1 rx=0x7f63fc047a90 tx=0x7f63fc04a7b0).stop 2021-03-09T10:56:33.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:33.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:33.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:33.026 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:33.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:56:33.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:33.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:33.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:33.027 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:33.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:33.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:33.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:33.028 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00661e0 con 0x7f63d8003400 2021-03-09T10:56:33.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:33.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079550 con 0x7f63d8009130 2021-03-09T10:56:33.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:33.029 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002edb0 con 0x7f63d8003840 2021-03-09T10:56:33.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:33.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:33.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:33.030 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:33.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:33.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:33.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:56:33.031 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:33.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.022+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:33.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:33.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:33.032 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:33.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:33.033 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:33.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:33.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:33.034 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:33.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:56:33.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:56:33.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:33.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:33.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d840 con 0x7f63d8003840 2021-03-09T10:56:33.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:33.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:33.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.023+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:56:33.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.024+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:56:33.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.024+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:33.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.024+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 secure :-1 s=READY pgs=21170 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f00733b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:33.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.024+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:56:33.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.024+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:33.038 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:33.032+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64563 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:33.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:56:33.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.032+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:33.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.032+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:33.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.032+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:33.039 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:33.040 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:33.040 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:33.040 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:33.040 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:33.041 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:33.041 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:33.041 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:33.041 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:33.041 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:33.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.033+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:56:33.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.033+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:33.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.033+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:56:33.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.034+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d30 con 0x7f63d8003840 2021-03-09T10:56:33.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.822+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:33.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:33.822+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:34.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:34.822+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:34.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:34.822+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:35.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:35.822+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:35.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:35.822+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:35.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:35.830+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:36.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:36.822+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:36.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:36.822+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:37.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:37.822+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:37.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:37.822+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 secure :-1 s=READY pgs=21170 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f00733b0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:38.035 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 secure :-1 s=READY pgs=21170 cs=0 l=1 rev1=1 rx=0x7f63f004de90 tx=0x7f63f00733b0).stop 2021-03-09T10:56:38.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:38.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:38.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006cdf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:38.036 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 unknown :-1 s=CLOSED pgs=21170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:56:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:38.037 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:38.038 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e420 con 0x7f63d8003400 2021-03-09T10:56:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:38.039 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bd40 con 0x7f63d8009130 2021-03-09T10:56:38.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:38.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:56:38.040 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.032+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:38.041 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.033+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:38.042 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.033+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:38.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.033+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.033+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:56:38.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.033+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:38.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.033+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:38.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:38.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:38.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:38.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:38.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:38.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:38.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:56:38.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:56:38.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:38.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:38.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007b940 con 0x7f63e0018390 2021-03-09T10:56:38.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:38.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:38.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000d840 con 0x7f63e0018390 2021-03-09T10:56:38.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:56:38.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.034+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:38.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.035+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:56:38.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.035+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:38.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.035+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21188 cs=0 l=1 rev1=1 rx=0x7f63e0008de0 tx=0x7f63f0006bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:38.055 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:38.041+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64662 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:38.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0011930 con 0x7f63e0018390 2021-03-09T10:56:38.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.042+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:38.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.042+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:38.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.042+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:38.058 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:38.058 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:38.058 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:38.059 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:38.059 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:38.060 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:38.061 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:38.061 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:38.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0011930 con 0x7f63e0018390 2021-03-09T10:56:38.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.042+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:38.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.042+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:56:38.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.043+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63e0018390 2021-03-09T10:56:38.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.822+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:38.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:38.822+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:39.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:39.823+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:39.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:39.823+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:40.824 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:40.823+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:40.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:40.823+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:40.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:40.823+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:40.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:40.831+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:41.823+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:41.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:41.823+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:42.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:42.823+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:42.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:42.823+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:43.044 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:56:43.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:43.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21188 cs=0 l=1 rev1=1 rx=0x7f63e0008de0 tx=0x7f63f0006bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:43.045 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21188 cs=0 l=1 rev1=1 rx=0x7f63e0008de0 tx=0x7f63f0006bc0).stop 2021-03-09T10:56:43.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:43.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:43.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:43.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:43.046 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:56:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:43.047 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:43.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:43.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:43.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:43.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079550 con 0x7f63d8003400 2021-03-09T10:56:43.048 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:43.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63d8009130 2021-03-09T10:56:43.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:43.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:56:43.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:43.049 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:43.050 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:43.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.042+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:43.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.043+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:43.051 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.043+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:43.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.043+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:56:43.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.043+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:43.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.043+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:43.052 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.044+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:43.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.044+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:43.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.044+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:43.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:43.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:43.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:43.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:43.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:43.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:56:43.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:56:43.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:43.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:43.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:56:43.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:43.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:43.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007b940 con 0x7f63d8003840 2021-03-09T10:56:43.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:56:43.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.045+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:43.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.046+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 secure :-1 s=READY pgs=21204 cs=0 l=1 rev1=1 rx=0x7f63f000d010 tx=0x7f63f00095c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:43.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.046+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:56:43.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.046+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:43.057 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:43.050+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64759 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:43.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00286c0 con 0x7f63d8003840 2021-03-09T10:56:43.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.051+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:43.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.051+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:43.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.051+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:43.059 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:43.060 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:43.060 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:43.060 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:43.060 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:43.060 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:43.061 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:43.061 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:43.061 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:43.061 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:43.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0011930 con 0x7f63d8003840 2021-03-09T10:56:43.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.051+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:43.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.051+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:56:43.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.053+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:56:43.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.823+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:43.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:43.823+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:44.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:44.823+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:44.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:44.823+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:45.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:45.823+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:45.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:45.823+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:45.832+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:46.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:46.823+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:46.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:46.823+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:47.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:47.824+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:47.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:47.824+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:48.053 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:48.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:48.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 secure :-1 s=READY pgs=21204 cs=0 l=1 rev1=1 rx=0x7f63f000d010 tx=0x7f63f00095c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:48.054 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 secure :-1 s=READY pgs=21204 cs=0 l=1 rev1=1 rx=0x7f63f000d010 tx=0x7f63f00095c0).stop 2021-03-09T10:56:48.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:48.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:48.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e006cdf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:48.055 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e006cdf0 unknown :-1 s=CLOSED pgs=21204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:48.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:56:48.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:48.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:48.056 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:48.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:48.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:48.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:48.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:48.057 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006bd40 con 0x7f63d8003400 2021-03-09T10:56:48.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:48.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63d8009130 2021-03-09T10:56:48.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:48.058 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005dea0 con 0x7f63e0018390 2021-03-09T10:56:48.059 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:48.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:48.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.051+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:48.060 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.052+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:48.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.052+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:48.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.052+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:48.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.052+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:56:48.061 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.052+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:48.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.052+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:48.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.053+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:48.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.053+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:48.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.053+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:48.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.053+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:48.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.053+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:48.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.053+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:48.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.053+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:48.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.053+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:48.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:56:48.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:56:48.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:48.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:48.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0017460 con 0x7f63e0018390 2021-03-09T10:56:48.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:48.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:48.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63e0018390 2021-03-09T10:56:48.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:56:48.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.054+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:48.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.055+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:56:48.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.055+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21217 cs=0 l=1 rev1=1 rx=0x7f63f0073380 tx=0x7f63f00382d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:48.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.055+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:48.067 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:48.059+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64861 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:48.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0038490 con 0x7f63e0018390 2021-03-09T10:56:48.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.059+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:48.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.059+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:48.069 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.059+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:48.069 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:48.069 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:48.069 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:48.070 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:48.070 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:48.070 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:48.070 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:48.071 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:48.071 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:48.071 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:48.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.060+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00286c0 con 0x7f63e0018390 2021-03-09T10:56:48.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.060+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:48.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.061+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:56:48.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.062+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63e0018390 2021-03-09T10:56:48.825 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.823+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:48.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:48.823+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:49.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:49.824+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:49.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:49.824+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:50.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:50.824+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:50.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:50.824+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:50.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:50.824+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:50.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:50.833+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:51.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:51.824+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:51.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:51.824+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:52.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:52.824+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:52.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:52.824+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:53.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:53.062 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21217 cs=0 l=1 rev1=1 rx=0x7f63f0073380 tx=0x7f63f00382d0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21217 cs=0 l=1 rev1=1 rx=0x7f63f0073380 tx=0x7f63f00382d0).stop 2021-03-09T10:56:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:53.063 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:53.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:53.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:56:53.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:53.064 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:53.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:53.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:53.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:53.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:53.065 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:53.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f63d8003400 2021-03-09T10:56:53.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:53.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8009130 2021-03-09T10:56:53.066 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:53.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:56:53.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.059+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:53.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.060+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:53.067 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.060+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:53.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.060+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:53.068 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.060+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:53.070 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.061+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:53.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.061+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:56:53.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.061+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:53.071 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.061+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:53.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.062+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:53.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.062+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:53.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.062+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:53.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:53.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:53.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:53.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:53.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:53.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:56:53.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:56:53.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:53.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:53.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00191a0 con 0x7f63d8003840 2021-03-09T10:56:53.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:53.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:53.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017460 con 0x7f63d8003840 2021-03-09T10:56:53.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:56:53.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:53.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 secure :-1 s=READY pgs=21233 cs=0 l=1 rev1=1 rx=0x7f63f0031860 tx=0x7f63f0013b30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:53.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:56:53.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.063+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:53.077 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:53.068+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 64961 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:53.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00316a0 con 0x7f63d8003840 2021-03-09T10:56:53.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.069+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:53.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.069+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:53.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.069+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:53.080 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:53.080 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:53.081 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:53.081 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:53.081 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:53.081 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:53.082 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:53.082 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:53.082 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:53.082 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:53.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.072+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00316a0 con 0x7f63d8003840 2021-03-09T10:56:53.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.072+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:53.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.072+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:56:53.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.073+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0029300 con 0x7f63d8003840 2021-03-09T10:56:53.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.824+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:53.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:53.824+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:54.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:54.824+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:54.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:54.824+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:55.826 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:55.825+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:55.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:55.825+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:55.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:55.834+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:56:56.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:56.825+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:56.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:56.825+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:57.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:57.825+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:57.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:57.825+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:58.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:56:58.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:56:58.072 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 secure :-1 s=READY pgs=21233 cs=0 l=1 rev1=1 rx=0x7f63f0031860 tx=0x7f63f0013b30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:56:58.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 secure :-1 s=READY pgs=21233 cs=0 l=1 rev1=1 rx=0x7f63f0031860 tx=0x7f63f0013b30).stop 2021-03-09T10:56:58.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:58.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:56:58.073 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:56:58.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 unknown :-1 s=CLOSED pgs=21233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:56:58.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:56:58.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:58.074 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:56:58.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:58.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:56:58.075 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:56:58.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:56:58.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:58.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0079550 con 0x7f63d8003400 2021-03-09T10:56:58.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:58.076 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63d8009130 2021-03-09T10:56:58.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:56:58.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002ac40 con 0x7f63e0018390 2021-03-09T10:56:58.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:56:58.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:56:58.077 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.069+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:58.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:56:58.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:56:58.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:56:58.078 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:56:58.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:56:58.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:56:58.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:56:58.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:56:58.079 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:56:58.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:56:58.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:58.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:58.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:56:58.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:56:58.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:56:58.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:56:58.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:56:58.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:58.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002bd60 con 0x7f63e0018390 2021-03-09T10:56:58.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:56:58.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:56:58.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0067990 con 0x7f63e0018390 2021-03-09T10:56:58.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:56:58.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.070+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:58.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.071+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21238 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc003a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:56:58.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.071+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:56:58.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.071+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:56:58.085 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:56:58.077+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65060 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:56:58.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc071070 con 0x7f63e0018390 2021-03-09T10:56:58.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.078+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:56:58.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.078+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:56:58.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.078+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:56:58.087 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:56:58.087 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:56:58.088 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:58.088 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:56:58.088 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:56:58.088 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:56:58.088 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:56:58.089 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:56:58.089 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:56:58.089 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:56:58.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc0711d0 con 0x7f63e0018390 2021-03-09T10:56:58.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.078+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:56:58.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.078+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:56:58.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.079+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc02b1d0 con 0x7f63e0018390 2021-03-09T10:56:58.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.825+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:58.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:58.825+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:56:59.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:59.825+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:56:59.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:56:59.825+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:00.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:00.825+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:00.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:00.825+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:00.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:00.825+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:00.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:00.835+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:01.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:01.825+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:01.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:01.825+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:02.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:02.826+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:02.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:02.826+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:03.081 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:03.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:03.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21238 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc003a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:03.082 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21238 cs=0 l=1 rev1=1 rx=0x7f63fc048af0 tx=0x7f63fc003a50).stop 2021-03-09T10:57:03.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:03.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:03.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:03.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:03.083 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:57:03.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:03.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:03.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:03.084 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:03.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:03.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:03.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:03.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011300 con 0x7f63d8003400 2021-03-09T10:57:03.085 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:03.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8009130 2021-03-09T10:57:03.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:03.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0014ce0 con 0x7f63d8003840 2021-03-09T10:57:03.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:03.086 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:03.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:03.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:03.087 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.078+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:03.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.079+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:03.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.079+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:57:03.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.079+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:03.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.079+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:03.088 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.079+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:03.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.079+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:03.089 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.079+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:03.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:03.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:03.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:03.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:03.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:03.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:57:03.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:57:03.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:03.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:03.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00138d0 con 0x7f63d8003840 2021-03-09T10:57:03.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:03.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:03.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002bd60 con 0x7f63d8003840 2021-03-09T10:57:03.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:57:03.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.080+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:03.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.081+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:57:03.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.081+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:03.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.081+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bef0 secure :-1 s=READY pgs=21253 cs=0 l=1 rev1=1 rx=0x7f63fc075fb0 tx=0x7f63fc051e00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:03.095 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:03.087+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65161 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:03.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052b40 con 0x7f63d8003840 2021-03-09T10:57:03.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.087+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:03.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.087+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:03.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.087+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:03.097 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:03.097 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:03.098 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:03.098 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:03.098 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:03.099 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:03.099 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:03.099 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:03.100 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:03.100 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:03.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.088+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc051a10 con 0x7f63d8003840 2021-03-09T10:57:03.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.088+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:03.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.088+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:57:03.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.089+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc052b40 con 0x7f63d8003840 2021-03-09T10:57:03.827 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.825+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:03.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:03.825+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:04.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:04.826+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:04.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:04.826+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:05.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:05.826+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:05.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:05.826+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:05.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:05.836+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:06.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:06.826+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:06.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:06.826+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:07.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:07.826+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:07.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:07.826+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:08.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:57:08.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:08.090 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bef0 secure :-1 s=READY pgs=21253 cs=0 l=1 rev1=1 rx=0x7f63fc075fb0 tx=0x7f63fc051e00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:08.091 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bef0 secure :-1 s=READY pgs=21253 cs=0 l=1 rev1=1 rx=0x7f63fc075fb0 tx=0x7f63fc051e00).stop 2021-03-09T10:57:08.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:08.092 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:08.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002bef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:08.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002bef0 unknown :-1 s=CLOSED pgs=21253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:08.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:57:08.093 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:08.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:08.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:08.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:08.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:08.094 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e0a0 con 0x7f63d8003400 2021-03-09T10:57:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:08.095 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:57:08.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:08.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63e0018390 2021-03-09T10:57:08.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.087+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:08.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.088+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:08.096 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.088+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:08.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.088+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:08.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.088+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:57:08.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.088+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:08.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.088+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:08.097 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:08.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:08.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:08.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:08.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:08.098 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:08.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:08.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:08.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:08.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.089+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:08.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:57:08.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:57:08.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:08.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:08.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001d210 con 0x7f63e0018390 2021-03-09T10:57:08.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:08.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:08.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0017e80 con 0x7f63e0018390 2021-03-09T10:57:08.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:57:08.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.090+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:08.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.091+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21270 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4005110).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:08.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.091+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:57:08.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.091+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:08.103 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:08.096+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65260 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:08.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40621f0 con 0x7f63e0018390 2021-03-09T10:57:08.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.097+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:08.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.097+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:08.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.097+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:08.105 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:08.105 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:08.105 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:08.106 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:08.106 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:08.106 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:08.107 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:08.107 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:08.107 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:08.107 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:08.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.100+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63e0018390 2021-03-09T10:57:08.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.100+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:08.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.100+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:57:08.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.102+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4048000 con 0x7f63e0018390 2021-03-09T10:57:08.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.826+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:08.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:08.826+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:09.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:09.826+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:09.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:09.826+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:10.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:10.827+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:10.827+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:10.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:10.827+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:10.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:10.837+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:11.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:11.827+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:11.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:11.827+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:12.828 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:12.826+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:12.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:12.826+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:13.099 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:13.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:13.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21270 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4005110).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:13.100 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21270 cs=0 l=1 rev1=1 rx=0x7f63f404b0d0 tx=0x7f63f4005110).stop 2021-03-09T10:57:13.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:13.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:13.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:13.101 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:13.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:57:13.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:13.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:13.102 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00018e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:13.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:13.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:13.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:13.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:13.103 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a930 con 0x7f63d8003400 2021-03-09T10:57:13.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:13.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:57:13.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:13.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002edb0 con 0x7f63d8003840 2021-03-09T10:57:13.104 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:13.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:13.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:13.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:13.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:13.105 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:13.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:57:13.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:13.106 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.097+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:13.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.098+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:13.107 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.098+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:13.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.098+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:13.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:13.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:13.108 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:13.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:13.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:13.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:57:13.109 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:57:13.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:13.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:13.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0019cc0 con 0x7f63d8003840 2021-03-09T10:57:13.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:13.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:13.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00138d0 con 0x7f63d8003840 2021-03-09T10:57:13.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:57:13.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.099+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:13.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.100+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00018e0 secure :-1 s=READY pgs=21285 cs=0 l=1 rev1=1 rx=0x7f63e0028230 tx=0x7f63fc013d60).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:13.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.100+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:57:13.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.100+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:13.113 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:13.107+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65360 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:13.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc078a40 con 0x7f63d8003840 2021-03-09T10:57:13.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.108+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:13.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.108+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:13.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.108+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:13.115 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:13.116 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:13.116 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:13.116 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:13.116 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:13.116 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:13.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.112+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63d8003840 2021-03-09T10:57:13.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.112+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:13.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.112+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:57:13.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.113+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc052430 con 0x7f63d8003840 2021-03-09T10:57:13.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.827+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:13.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:13.827+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:14.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:14.827+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:14.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:14.827+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:15.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:15.827+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:15.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:15.827+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:15.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:15.838+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:16.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:16.827+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:16.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:16.827+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:17.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:17.828+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:17.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:17.828+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:18.110 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:18.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:18.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00018e0 secure :-1 s=READY pgs=21285 cs=0 l=1 rev1=1 rx=0x7f63e0028230 tx=0x7f63fc013d60).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:18.111 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00018e0 secure :-1 s=READY pgs=21285 cs=0 l=1 rev1=1 rx=0x7f63e0028230 tx=0x7f63fc013d60).stop 2021-03-09T10:57:18.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:18.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:18.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e00018e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:18.112 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e00018e0 unknown :-1 s=CLOSED pgs=21285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:18.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:57:18.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:18.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:18.113 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:18.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:18.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:18.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:18.114 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:18.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:57:18.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:18.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0068560 con 0x7f63d8009130 2021-03-09T10:57:18.115 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:18.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63e0018390 2021-03-09T10:57:18.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:18.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:18.116 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:18.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:18.117 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.108+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:18.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.109+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:18.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.109+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:57:18.118 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.109+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:18.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.109+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:18.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.109+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:18.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.109+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:18.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.109+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:18.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.110+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:18.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:18.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:18.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:18.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:18.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:57:18.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:57:18.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:18.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:18.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000c570 con 0x7f63e0018390 2021-03-09T10:57:18.122 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:18.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:18.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0019cc0 con 0x7f63e0018390 2021-03-09T10:57:18.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:57:18.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:18.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:57:18.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:18.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.111+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21296 cs=0 l=1 rev1=1 rx=0x7f63e00175f0 tx=0x7f63fc023ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:18.125 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:18.117+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65459 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:18.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc051e70 con 0x7f63e0018390 2021-03-09T10:57:18.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.118+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:18.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.118+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:18.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.118+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:18.127 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:18.127 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:18.127 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:18.127 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:18.127 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:18.128 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:18.128 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:18.128 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:18.128 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:18.129 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:18.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.121+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc078a40 con 0x7f63e0018390 2021-03-09T10:57:18.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.121+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:18.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.121+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:57:18.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.122+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:57:18.829 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.828+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:18.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:18.828+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:19.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:19.827+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:19.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:19.827+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:20.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:20.827+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:20.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:20.827+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:20.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:20.828+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:20.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:20.839+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:21.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:21.828+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:21.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:21.828+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:22.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:22.828+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:22.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:22.828+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:23.119 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:23.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:23.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21296 cs=0 l=1 rev1=1 rx=0x7f63e00175f0 tx=0x7f63fc023ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:23.120 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21296 cs=0 l=1 rev1=1 rx=0x7f63e00175f0 tx=0x7f63fc023ed0).stop 2021-03-09T10:57:23.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:23.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:23.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:23.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:23.121 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:57:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:23.123 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:23.124 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e0a0 con 0x7f63d8003400 2021-03-09T10:57:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:57:23.125 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:57:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:23.126 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:23.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:23.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.118+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:23.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.119+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:23.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.119+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:57:23.127 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.119+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:23.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.119+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:23.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.119+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:23.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.119+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:23.128 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.119+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:23.129 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:57:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:57:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:23.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:57:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e000c570 con 0x7f63d8003840 2021-03-09T10:57:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:57:23.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.120+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:23.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.121+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:57:23.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.121+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 secure :-1 s=READY pgs=21312 cs=0 l=1 rev1=1 rx=0x7f63fc046b20 tx=0x7f63fc03cf90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:23.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.121+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:23.132 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:23.126+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65559 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:23.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc052260 con 0x7f63d8003840 2021-03-09T10:57:23.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.127+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:23.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.127+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:23.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.127+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:23.135 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:23.135 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:23.135 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:23.135 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:23.136 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:23.136 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:23.136 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:23.136 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:23.136 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:23.137 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:23.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065790 con 0x7f63d8003840 2021-03-09T10:57:23.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.129+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:23.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.129+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:57:23.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.129+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63d8003840 2021-03-09T10:57:23.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.828+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:23.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:23.828+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:24.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:24.829+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:24.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:24.829+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:25.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:25.829+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:25.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:25.829+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:25.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:25.840+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:26.830 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:26.828+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:26.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:26.828+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:27.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:27.829+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:27.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:27.829+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:28.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:28.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:28.130 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 secure :-1 s=READY pgs=21312 cs=0 l=1 rev1=1 rx=0x7f63fc046b20 tx=0x7f63fc03cf90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:28.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 secure :-1 s=READY pgs=21312 cs=0 l=1 rev1=1 rx=0x7f63fc046b20 tx=0x7f63fc03cf90).stop 2021-03-09T10:57:28.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:28.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:28.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:28.131 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 unknown :-1 s=CLOSED pgs=21312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:28.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:57:28.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:28.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:28.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:28.132 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:28.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:28.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:28.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:28.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003400 2021-03-09T10:57:28.133 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:28.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065290 con 0x7f63d8009130 2021-03-09T10:57:28.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:28.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63e0018390 2021-03-09T10:57:28.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:28.134 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.127+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:28.135 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:57:28.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:28.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:28.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:28.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:28.136 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:28.137 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:28.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:28.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:28.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:28.138 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:28.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:57:28.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:57:28.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:28.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:28.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079500 con 0x7f63e0018390 2021-03-09T10:57:28.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:28.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:28.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.128+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63e0018390 2021-03-09T10:57:28.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.129+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:57:28.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.129+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:28.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.129+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21323 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc050fb0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:28.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.129+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:57:28.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.129+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:28.142 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:28.136+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65656 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:28.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc06f290 con 0x7f63e0018390 2021-03-09T10:57:28.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.137+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:28.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.137+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:28.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.137+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:28.143 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:28.144 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:28.144 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:28.144 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:28.144 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:28.144 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:28.145 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:28.145 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:28.145 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:28.145 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:28.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.138+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc065790 con 0x7f63e0018390 2021-03-09T10:57:28.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.138+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:28.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.138+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:57:28.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.139+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc078c00 con 0x7f63e0018390 2021-03-09T10:57:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.829+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:28.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:28.829+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:29.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:29.829+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:29.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:29.829+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:30.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:30.829+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:30.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:30.829+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:30.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:30.829+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:30.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:30.841+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:31.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:31.830+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:31.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:31.830+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:32.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:32.830+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:32.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:32.830+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:33.139 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:33.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:33.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21323 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc050fb0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:33.140 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21323 cs=0 l=1 rev1=1 rx=0x7f63fc011000 tx=0x7f63fc050fb0).stop 2021-03-09T10:57:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:33.141 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:57:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:33.142 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:57:33.143 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:33.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0065fa0 con 0x7f63d8009130 2021-03-09T10:57:33.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:33.144 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:57:33.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:33.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.137+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:33.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.138+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:33.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.138+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:33.145 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.138+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:33.146 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.138+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:33.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.138+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:57:33.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.138+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:33.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.138+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:33.147 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:33.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:33.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:33.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:33.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:33.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:33.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:33.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:33.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:57:33.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:57:33.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:33.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:33.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0026510 con 0x7f63d8003840 2021-03-09T10:57:33.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:33.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:33.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.139+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001d210 con 0x7f63d8003840 2021-03-09T10:57:33.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.140+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:57:33.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.140+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:33.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.140+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 secure :-1 s=READY pgs=21336 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4017ed0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:33.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.140+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:57:33.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.140+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:33.152 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:33.146+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65759 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:33.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63d8003840 2021-03-09T10:57:33.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.147+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:33.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.147+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:33.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.147+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:33.154 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:33.154 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:33.154 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:33.154 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:33.155 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:33.155 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:33.155 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:33.155 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:33.155 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:33.156 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:33.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.151+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4021170 con 0x7f63d8003840 2021-03-09T10:57:33.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.151+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:33.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.151+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:57:33.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.152+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63d8003840 2021-03-09T10:57:33.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.830+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:33.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:33.830+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:34.831 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:34.830+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:34.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:34.830+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:35.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:35.830+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:35.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:35.830+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:35.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:35.842+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:36.830+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:36.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:36.830+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:37.830+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:37.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:37.830+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:38.148 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:38.149 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:38.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 secure :-1 s=READY pgs=21336 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4017ed0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:38.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 secure :-1 s=READY pgs=21336 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4017ed0).stop 2021-03-09T10:57:38.150 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:38.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:38.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e002b870 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:38.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e002b870 unknown :-1 s=CLOSED pgs=21336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:38.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:57:38.151 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:38.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:38.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:38.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:38.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:38.152 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:38.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:38.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0011fb0 con 0x7f63d8003400 2021-03-09T10:57:38.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:38.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b500 con 0x7f63d8009130 2021-03-09T10:57:38.153 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:38.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63e0018390 2021-03-09T10:57:38.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:38.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:38.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:38.154 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:38.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.147+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:38.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:38.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:57:38.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:38.155 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:38.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:38.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:38.156 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:38.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:38.157 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:38.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:38.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:38.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:38.158 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:57:38.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:57:38.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:38.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:38.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007a2f0 con 0x7f63e0018390 2021-03-09T10:57:38.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:38.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:38.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0026510 con 0x7f63e0018390 2021-03-09T10:57:38.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:57:38.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.148+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:38.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.149+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21350 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f402d2a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:38.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.149+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:57:38.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.149+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:38.161 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:38.156+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65858 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:38.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.156+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63e0018390 2021-03-09T10:57:38.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.156+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:38.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.156+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:38.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.156+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:38.163 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:38.163 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:38.164 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:38.164 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:38.164 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:38.164 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:38.164 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:38.165 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:38.165 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:38.165 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:38.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63e0018390 2021-03-09T10:57:38.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.157+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.158+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:57:38.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.158+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4022a80 con 0x7f63e0018390 2021-03-09T10:57:38.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.830+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:38.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:38.830+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:39.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:39.831+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:39.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:39.831+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:40.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:40.831+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:40.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:40.831+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:40.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:40.831+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:40.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:40.843+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:41.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:41.831+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:41.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:41.831+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:42.832 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:42.831+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:42.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:42.831+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:43.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:43.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:43.159 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21350 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f402d2a0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:43.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 secure :-1 s=READY pgs=21350 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f402d2a0).stop 2021-03-09T10:57:43.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:43.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:43.160 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0078f70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:43.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0078f70 unknown :-1 s=CLOSED pgs=21350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:43.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:57:43.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:43.161 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:43.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:43.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:43.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:43.162 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:43.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:43.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:57:43.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:43.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0017ed0 con 0x7f63d8009130 2021-03-09T10:57:43.163 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002edb0 con 0x7f63d8003840 2021-03-09T10:57:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:43.164 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:43.165 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:57:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:43.166 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:43.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:43.167 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:43.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.157+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:43.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:43.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:43.168 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:43.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:43.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:43.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:57:43.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:57:43.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e007a160 con 0x7f63d8003840 2021-03-09T10:57:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:43.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079500 con 0x7f63d8003840 2021-03-09T10:57:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:57:43.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:57:43.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:43.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.159+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21369 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc065870).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:43.174 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:43.166+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 65958 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:43.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.167+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0488e0 con 0x7f63d8003840 2021-03-09T10:57:43.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.167+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:43.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.167+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:43.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.167+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:43.177 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:43.177 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:43.177 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:43.178 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:43.178 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:43.179 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:43.179 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:43.179 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:43.180 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:43.180 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:43.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.170+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc00b3c0 con 0x7f63d8003840 2021-03-09T10:57:43.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.170+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:43.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.170+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:57:43.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.172+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc029290 con 0x7f63d8003840 2021-03-09T10:57:43.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.830+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:43.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:43.830+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:44.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:44.831+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:44.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:44.831+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:45.831+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:45.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:45.831+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:45.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:45.844+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:46.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:46.832+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:46.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:46.832+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:47.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:47.832+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:47.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:47.832+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:48.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:57:48.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:48.169 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21369 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc065870).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:48.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21369 cs=0 l=1 rev1=1 rx=0x7f63fc000f80 tx=0x7f63fc065870).stop 2021-03-09T10:57:48.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:48.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:48.170 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:48.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=CLOSED pgs=21369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:48.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:57:48.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:48.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:48.171 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:48.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:48.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:48.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:48.172 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:48.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003400 2021-03-09T10:57:48.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:48.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63d8009130 2021-03-09T10:57:48.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:48.173 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b500 con 0x7f63e0018390 2021-03-09T10:57:48.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.166+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:48.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:48.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:48.174 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:48.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:48.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:48.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:57:48.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:48.175 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:48.176 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:48.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:48.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.167+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:48.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:48.177 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:48.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:48.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:48.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:48.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:57:48.178 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:57:48.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:48.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:48.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0013210 con 0x7f63e0018390 2021-03-09T10:57:48.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:48.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:48.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007a160 con 0x7f63e0018390 2021-03-09T10:57:48.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:57:48.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:48.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:57:48.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:48.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.168+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21376 cs=0 l=1 rev1=1 rx=0x7f63fc03afd0 tx=0x7f63fc0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:48.181 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:48.176+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66057 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:48.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc050590 con 0x7f63e0018390 2021-03-09T10:57:48.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.177+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:48.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.177+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:48.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.177+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:48.183 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:48.183 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:48.183 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:48.184 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:48.184 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:48.184 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:48.184 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:48.185 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:48.185 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:48.185 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:48.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.178+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc050590 con 0x7f63e0018390 2021-03-09T10:57:48.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.178+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:48.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.178+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:57:48.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.179+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc047270 con 0x7f63e0018390 2021-03-09T10:57:48.833 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.831+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:48.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:48.831+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:49.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:49.832+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:49.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:49.832+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:50.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:50.832+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:50.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:50.832+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:50.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:50.832+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:50.846 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:50.845+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:51.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:51.832+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:51.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:51.832+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:52.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:52.832+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:52.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:52.832+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:53.179 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21376 cs=0 l=1 rev1=1 rx=0x7f63fc03afd0 tx=0x7f63fc0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21376 cs=0 l=1 rev1=1 rx=0x7f63fc03afd0 tx=0x7f63fc0512e0).stop 2021-03-09T10:57:53.180 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:53.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:53.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:53.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=CLOSED pgs=21376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:53.181 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:57:53.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:53.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:53.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:53.182 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:57:53.183 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015bf0 con 0x7f63d8009130 2021-03-09T10:57:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003840 2021-03-09T10:57:53.184 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:53.185 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.177+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:53.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.178+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:57:53.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.178+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:53.186 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.178+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:53.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.178+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:53.187 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.178+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:53.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.178+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:53.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:53.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:53.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:53.188 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:53.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:53.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:57:53.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:57:53.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:53.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:53.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b610 con 0x7f63d8003840 2021-03-09T10:57:53.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:53.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00191a0 con 0x7f63d8003840 2021-03-09T10:57:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:57:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.179+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:53.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.180+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21396 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004dd90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:53.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.180+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:57:53.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.180+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:53.192 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:53.186+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66157 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:53.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:57:53.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.187+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:53.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.187+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:53.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.187+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:53.194 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:53.194 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:53.194 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:53.195 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:53.195 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:53.195 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:53.195 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:53.195 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:53.196 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:53.196 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:53.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.190+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:57:53.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.190+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:53.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.190+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:57:53.197 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.192+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ed20 con 0x7f63d8003840 2021-03-09T10:57:53.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.832+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:53.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:53.832+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:54.832+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:54.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:54.832+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:55.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:55.833+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:55.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:55.833+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:55.847 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:55.846+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:57:56.834 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:56.833+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:56.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:56.833+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:57.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:57.833+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:57.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:57.833+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:58.189 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:57:58.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:57:58.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21396 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004dd90).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:57:58.190 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21396 cs=0 l=1 rev1=1 rx=0x7f63f0000ea0 tx=0x7f63f004dd90).stop 2021-03-09T10:57:58.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:58.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:57:58.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:57:58.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=CLOSED pgs=21396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:57:58.191 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:57:58.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:58.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:57:58.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:58.192 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:57:58.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:57:58.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:57:58.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:58.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003400 2021-03-09T10:57:58.193 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:58.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0025460 con 0x7f63d8009130 2021-03-09T10:57:58.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:57:58.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b500 con 0x7f63e0018390 2021-03-09T10:57:58.194 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:57:58.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:57:58.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:58.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:57:58.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:57:58.195 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:57:58.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:57:58.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:57:58.196 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.187+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:57:58.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.188+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:57:58.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.188+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:57:58.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.188+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:57:58.198 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:57:58.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:58.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:58.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:57:58.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:57:58.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:57:58.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:57:58.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:57:58.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:58.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024c30 con 0x7f63e0018390 2021-03-09T10:57:58.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:57:58.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:57:58.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0013210 con 0x7f63e0018390 2021-03-09T10:57:58.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:57:58.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.189+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:58.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.190+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21409 cs=0 l=1 rev1=1 rx=0x7f63e0038350 tx=0x7f63fc078fe0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:57:58.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.190+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:57:58.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.190+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:57:58.203 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:57:58.195+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66256 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:57:58.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc078880 con 0x7f63e0018390 2021-03-09T10:57:58.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.196+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:57:58.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.196+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:57:58.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.196+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:57:58.204 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:57:58.204 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:57:58.205 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:58.205 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:57:58.205 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:57:58.206 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:57:58.206 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:57:58.206 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:57:58.206 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:57:58.207 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:57:58.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.202+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc078880 con 0x7f63e0018390 2021-03-09T10:57:58.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.202+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:57:58.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.202+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:57:58.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.203+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc051560 con 0x7f63e0018390 2021-03-09T10:57:58.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.832+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:58.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:58.832+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:57:59.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:59.833+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:57:59.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:57:59.833+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:00.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:00.833+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:00.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:00.833+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:00.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:00.833+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:00.848 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:00.847+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:01.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:01.834+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:01.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:01.834+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:02.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:02.834+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:02.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:02.834+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:03.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:03.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:03.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21409 cs=0 l=1 rev1=1 rx=0x7f63e0038350 tx=0x7f63fc078fe0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:03.199 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21409 cs=0 l=1 rev1=1 rx=0x7f63e0038350 tx=0x7f63fc078fe0).stop 2021-03-09T10:58:03.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:03.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:03.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:03.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=CLOSED pgs=21409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:03.200 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:58:03.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:03.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:03.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:03.201 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:03.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:03.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:03.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:03.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63d8003400 2021-03-09T10:58:03.202 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:03.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005ea90 con 0x7f63d8009130 2021-03-09T10:58:03.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:03.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003840 2021-03-09T10:58:03.203 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:03.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:03.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.196+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:03.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.197+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:03.204 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.197+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:03.205 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.197+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:03.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.197+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:58:03.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.197+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:03.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.197+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:03.206 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.197+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:03.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.197+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:03.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.197+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:03.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:03.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:03.207 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:03.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:03.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:03.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:58:03.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:58:03.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:03.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:03.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005d400 con 0x7f63d8003840 2021-03-09T10:58:03.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:03.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:03.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b610 con 0x7f63d8003840 2021-03-09T10:58:03.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:58:03.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:03.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21417 cs=0 l=1 rev1=1 rx=0x7f63f004d290 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:03.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:58:03.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.198+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:03.211 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:03.205+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66356 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:03.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0006bc0 con 0x7f63d8003840 2021-03-09T10:58:03.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.206+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:03.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.206+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:03.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.206+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:03.213 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:03.213 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:03.214 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:03.214 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:03.214 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:03.214 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:03.214 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:03.215 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:03.215 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:03.215 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:03.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0030220 con 0x7f63d8003840 2021-03-09T10:58:03.215 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.206+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:03.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.206+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:58:03.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.207+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0055290 con 0x7f63d8003840 2021-03-09T10:58:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.834+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:03.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:03.834+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:04.835 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:04.834+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:04.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:04.834+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:05.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:05.834+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:05.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:05.834+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:05.850 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:05.848+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:06.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:06.834+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:06.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:06.834+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:07.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:07.834+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:07.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:07.834+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:08.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:08.208 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:08.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21417 cs=0 l=1 rev1=1 rx=0x7f63f004d290 tx=0x7f63f004db20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:08.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21417 cs=0 l=1 rev1=1 rx=0x7f63f004d290 tx=0x7f63f004db20).stop 2021-03-09T10:58:08.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:08.209 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:08.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:08.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=CLOSED pgs=21417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:08.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:58:08.210 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:08.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:08.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:08.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:08.211 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:08.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:08.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:08.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006cbf0 con 0x7f63d8003400 2021-03-09T10:58:08.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:08.212 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8009130 2021-03-09T10:58:08.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:08.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63e0018390 2021-03-09T10:58:08.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:08.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:08.213 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:08.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:08.214 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.206+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:08.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.207+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:08.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.207+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:58:08.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.207+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:08.216 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.207+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:08.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.207+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:08.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.207+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:08.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.207+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:08.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:08.217 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:08.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:08.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:08.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:08.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:58:08.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:58:08.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:08.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:08.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079500 con 0x7f63e0018390 2021-03-09T10:58:08.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:08.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:08.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024c30 con 0x7f63e0018390 2021-03-09T10:58:08.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:58:08.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.208+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:08.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.209+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:58:08.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.210+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:08.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.209+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21436 cs=0 l=1 rev1=1 rx=0x7f63e006a7f0 tx=0x7f63fc06de00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:08.221 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:08.215+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66455 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:08.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:58:08.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.216+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:08.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.216+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:08.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.216+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:08.223 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:08.224 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:08.224 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:08.224 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:08.224 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:08.224 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:08.225 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:08.225 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:08.225 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:08.225 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:08.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.217+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc05ddb0 con 0x7f63e0018390 2021-03-09T10:58:08.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.217+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:08.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.219+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63e0018390 2021-03-09T10:58:08.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.220+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:58:08.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.834+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:08.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:08.834+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:09.835+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:09.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:09.835+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:10.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:10.835+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:10.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:10.835+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:10.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:10.835+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:10.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:10.849+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:11.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:11.835+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:11.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:11.835+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:12.836 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:12.835+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:12.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:12.835+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:13.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:13.218 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:13.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21436 cs=0 l=1 rev1=1 rx=0x7f63e006a7f0 tx=0x7f63fc06de00).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:13.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21436 cs=0 l=1 rev1=1 rx=0x7f63e006a7f0 tx=0x7f63fc06de00).stop 2021-03-09T10:58:13.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:13.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:13.219 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:13.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=CLOSED pgs=21436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:13.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:58:13.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:13.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:13.220 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:13.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:13.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:13.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:13.221 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:13.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b500 con 0x7f63d8003400 2021-03-09T10:58:13.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:13.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0010700 con 0x7f63d8009130 2021-03-09T10:58:13.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:13.222 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e320 con 0x7f63d8003840 2021-03-09T10:58:13.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:13.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:13.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:13.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:13.223 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:13.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:13.225 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:58:13.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:13.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.216+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:13.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:13.226 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:13.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:13.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:13.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:13.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:13.227 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:13.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:13.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:58:13.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:58:13.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:13.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:13.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006be40 con 0x7f63d8003840 2021-03-09T10:58:13.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:13.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:13.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0079500 con 0x7f63d8003840 2021-03-09T10:58:13.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:58:13.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.217+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:13.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.218+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21445 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc078ec0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:13.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.218+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:58:13.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.218+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:13.231 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:13.225+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66555 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:13.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.225+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8003840 2021-03-09T10:58:13.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.225+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:13.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.225+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:13.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.225+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:13.233 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:13.233 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:13.233 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:13.233 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:13.233 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:13.234 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:13.234 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:13.234 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:13.234 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:13.235 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:13.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.229+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc06fe20 con 0x7f63d8003840 2021-03-09T10:58:13.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.229+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:13.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.229+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:58:13.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.230+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc065240 con 0x7f63d8003840 2021-03-09T10:58:13.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.835+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:13.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:13.835+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:14.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:14.835+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:14.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:14.835+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:15.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:15.836+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:15.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:15.836+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:15.851 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:15.850+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:16.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:16.836+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:16.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:16.836+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:17.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:17.836+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:17.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:17.836+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:18.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:18.228 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:18.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21445 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc078ec0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:18.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21445 cs=0 l=1 rev1=1 rx=0x7f63fc075160 tx=0x7f63fc078ec0).stop 2021-03-09T10:58:18.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:18.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:18.229 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:18.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=CLOSED pgs=21445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:18.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,2,1] 2021-03-09T10:58:18.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:18.230 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:18.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:18.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:18.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:18.231 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:18.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:18.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0000c70 con 0x7f63d8003400 2021-03-09T10:58:18.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:18.232 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0024e80 con 0x7f63d8009130 2021-03-09T10:58:18.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:18.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b700 con 0x7f63e0018390 2021-03-09T10:58:18.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:18.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:18.233 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:18.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:18.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:18.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:18.234 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:58:18.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:18.235 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:18.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.226+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:18.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:18.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:18.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:58:18.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:58:18.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:18.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:18.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0024c30 con 0x7f63e0018390 2021-03-09T10:58:18.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:18.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:18.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006be40 con 0x7f63e0018390 2021-03-09T10:58:18.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:58:18.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:18.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:58:18.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21458 cs=0 l=1 rev1=1 rx=0x7f63e0066bc0 tx=0x7f63fc047400).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:18.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.227+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:18.241 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:18.234+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66654 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:18.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.234+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc049700 con 0x7f63e0018390 2021-03-09T10:58:18.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.234+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:18.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.234+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:18.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.234+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:18.242 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:18.243 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:18.243 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:18.243 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:18.243 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:18.244 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:18.244 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:18.244 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:18.244 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:18.245 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:18.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc047470 con 0x7f63e0018390 2021-03-09T10:58:18.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.237+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:18.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.237+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:58:18.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.237+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049700 con 0x7f63e0018390 2021-03-09T10:58:18.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.836+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:18.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:18.836+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:19.837 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:19.836+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:19.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:19.836+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:20.836+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:20.836+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:20.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:20.836+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:20.852 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:20.851+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:21.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:21.836+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:21.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:21.836+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:22.837+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:22.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:22.837+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:23.236 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:23.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:23.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21458 cs=0 l=1 rev1=1 rx=0x7f63e0066bc0 tx=0x7f63fc047400).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:23.237 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 secure :-1 s=READY pgs=21458 cs=0 l=1 rev1=1 rx=0x7f63e0066bc0 tx=0x7f63fc047400).stop 2021-03-09T10:58:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0077720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0077720 unknown :-1 s=CLOSED pgs=21458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:23.238 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:58:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:23.239 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0027e90 con 0x7f63d8003400 2021-03-09T10:58:23.240 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:58:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:23.241 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00193f0 con 0x7f63d8003840 2021-03-09T10:58:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:23.242 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:23.243 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:58:23.244 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:23.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:23.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:23.245 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:23.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.235+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:23.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:23.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:23.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:58:23.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:58:23.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:23.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:23.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e001b610 con 0x7f63d8003840 2021-03-09T10:58:23.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:23.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:23.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e0024c30 con 0x7f63d8003840 2021-03-09T10:58:23.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:58:23.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:23.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:58:23.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21473 cs=0 l=1 rev1=1 rx=0x7f63e002c530 tx=0x7f63fc0512e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:23.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.236+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:23.251 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:23.243+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66754 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:23.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63d8003840 2021-03-09T10:58:23.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.243+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:23.253 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.243+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:23.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.243+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:23.254 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:23.254 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:23.255 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:23.256 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:23.256 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:23.256 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:23.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.243+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02c830 con 0x7f63d8003840 2021-03-09T10:58:23.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.243+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:23.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.243+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:58:23.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc049700 con 0x7f63d8003840 2021-03-09T10:58:23.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.837+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:23.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:23.837+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:24.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:24.837+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:24.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:24.837+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:25.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:25.837+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:25.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:25.837+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:25.853 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:25.852+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:26.838 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:26.836+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:26.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:26.836+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:27.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:27.837+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:27.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:27.837+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:28.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2021-03-09T10:58:28.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:28.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21473 cs=0 l=1 rev1=1 rx=0x7f63e002c530 tx=0x7f63fc0512e0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:28.246 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21473 cs=0 l=1 rev1=1 rx=0x7f63e002c530 tx=0x7f63fc0512e0).stop 2021-03-09T10:58:28.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:28.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:28.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:28.247 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=CLOSED pgs=21473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:28.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:58:28.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:28.248 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:28.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:28.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:28.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0022230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:28.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:28.249 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:28.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e320 con 0x7f63d8003400 2021-03-09T10:58:28.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:28.250 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0008bd0 con 0x7f63d8009130 2021-03-09T10:58:28.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:28.251 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0013490 con 0x7f63e0018390 2021-03-09T10:58:28.252 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:28.254 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:28.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:28.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:28.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:28.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0022230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:28.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:58:28.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:28.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:28.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:28.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:28.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.244+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:28.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:28.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:28.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:28.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:28.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:28.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:58:28.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:58:28.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:28.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:28.262 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00679d0 con 0x7f63e0018390 2021-03-09T10:58:28.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:28.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:28.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005d400 con 0x7f63e0018390 2021-03-09T10:58:28.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:58:28.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.245+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:28.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.246+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0022230 secure :-1 s=READY pgs=21487 cs=0 l=1 rev1=1 rx=0x7f63e004e0a0 tx=0x7f63f0000bc0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:28.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.246+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:58:28.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.246+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:28.266 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:28.253+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66853 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:28.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.253+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0090530 con 0x7f63e0018390 2021-03-09T10:58:28.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.253+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:28.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.253+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:28.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.253+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:28.269 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:28.269 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:28.269 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:28.270 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:28.270 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:28.271 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:28.271 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:28.271 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:28.272 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:28.272 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:28.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.253+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0090350 con 0x7f63e0018390 2021-03-09T10:58:28.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.253+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:28.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.253+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:58:28.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f007ed20 con 0x7f63e0018390 2021-03-09T10:58:28.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.836+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:28.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:28.836+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:29.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:29.838+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:29.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:29.838+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:30.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:30.838+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:30.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:30.838+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:30.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:30.838+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:30.854 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:30.853+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:31.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:31.838+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:31.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:31.838+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:32.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:32.838+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:32.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:32.838+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:33.255 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0022230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0022230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0022230 secure :-1 s=READY pgs=21487 cs=0 l=1 rev1=1 rx=0x7f63e004e0a0 tx=0x7f63f0000bc0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:33.256 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0022230 secure :-1 s=READY pgs=21487 cs=0 l=1 rev1=1 rx=0x7f63e004e0a0 tx=0x7f63f0000bc0).stop 2021-03-09T10:58:33.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:33.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:33.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0022230 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:33.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0022230 unknown :-1 s=CLOSED pgs=21487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:33.257 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:58:33.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:33.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:33.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:33.258 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006b700 con 0x7f63d8003400 2021-03-09T10:58:33.259 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:33.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:58:33.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:33.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0022c40 con 0x7f63d8003840 2021-03-09T10:58:33.260 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:33.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:33.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:33.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:33.261 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:33.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:33.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:58:33.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:33.263 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.254+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:33.264 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:33.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:33.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:33.265 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:33.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:33.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:58:33.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:58:33.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:33.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:33.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00666d0 con 0x7f63d8003840 2021-03-09T10:58:33.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:33.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:33.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e007a2f0 con 0x7f63d8003840 2021-03-09T10:58:33.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:58:33.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.255+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:33.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.256+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:58:33.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.256+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:33.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.256+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21506 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f40036c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:33.269 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:33.264+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 66953 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:33.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.264+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40223d0 con 0x7f63d8003840 2021-03-09T10:58:33.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.264+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:33.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.264+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:33.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.264+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:33.271 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:33.271 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:33.271 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:33.272 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:33.272 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:33.272 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:33.272 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:33.273 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:33.273 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:33.273 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:33.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.268+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40223d0 con 0x7f63d8003840 2021-03-09T10:58:33.273 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.268+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:33.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.268+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:58:33.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.269+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f40461f0 con 0x7f63d8003840 2021-03-09T10:58:33.839 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.838+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:33.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:33.838+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:34.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:34.838+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:34.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:34.838+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:35.838+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:35.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:35.838+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:35.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:35.854+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:36.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:36.839+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:36.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:36.839+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:37.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:37.839+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:37.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:37.839+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:38.266 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.264+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:38.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.264+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:38.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.264+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21506 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f40036c0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:38.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.264+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21506 cs=0 l=1 rev1=1 rx=0x7f63f4017ea0 tx=0x7f63f40036c0).stop 2021-03-09T10:58:38.267 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.264+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:38.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.264+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:38.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.264+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:38.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=CLOSED pgs=21506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:38.268 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,0,2] 2021-03-09T10:58:38.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:38.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:38.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:38.269 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:38.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00174c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:38.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:38.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:38.270 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cd10 con 0x7f63d8003400 2021-03-09T10:58:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a8f0 con 0x7f63d8009130 2021-03-09T10:58:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005e420 con 0x7f63e0018390 2021-03-09T10:58:38.271 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:38.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:38.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:38.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:38.272 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.265+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:38.274 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.266+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00174c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:38.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.266+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:58:38.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.266+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:38.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.266+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:38.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.267+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:38.275 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.267+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:38.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.267+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:38.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:38.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:38.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:38.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:38.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:38.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:58:38.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:58:38.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:38.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:38.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e002d600 con 0x7f63e0018390 2021-03-09T10:58:38.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:38.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:38.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00679d0 con 0x7f63e0018390 2021-03-09T10:58:38.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:58:38.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:38.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:58:38.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:38.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.268+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00174c0 secure :-1 s=READY pgs=21524 cs=0 l=1 rev1=1 rx=0x7f63f0028870 tx=0x7f63f0028720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:38.281 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:38.274+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 67052 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:38.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0028480 con 0x7f63e0018390 2021-03-09T10:58:38.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.274+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:38.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.274+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.274+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:38.283 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:38.284 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:38.284 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:38.284 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:38.284 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:38.285 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.275+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63e0018390 2021-03-09T10:58:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.275+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:38.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.276+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:58:38.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.276+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f005e7a0 con 0x7f63e0018390 2021-03-09T10:58:38.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.839+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:38.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:38.839+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:39.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:39.839+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:39.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:39.839+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:40.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:40.839+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:40.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:40.839+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:40.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:40.839+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:40.855 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:40.855+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:41.840 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:41.839+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:41.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:41.839+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:42.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:42.839+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:42.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:42.839+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:43.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00174c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:43.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00174c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:43.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00174c0 secure :-1 s=READY pgs=21524 cs=0 l=1 rev1=1 rx=0x7f63f0028870 tx=0x7f63f0028720).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:43.276 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00174c0 secure :-1 s=READY pgs=21524 cs=0 l=1 rev1=1 rx=0x7f63f0028870 tx=0x7f63f0028720).stop 2021-03-09T10:58:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00174c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00174c0 unknown :-1 s=CLOSED pgs=21524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:43.277 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:58:43.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:43.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:43.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:43.278 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:43.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:43.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:43.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:43.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002f280 con 0x7f63d8003400 2021-03-09T10:58:43.279 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:43.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0077160 con 0x7f63d8009130 2021-03-09T10:58:43.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:43.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00193f0 con 0x7f63d8003840 2021-03-09T10:58:43.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:43.280 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:43.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:43.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.274+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:43.281 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.275+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:43.282 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.276+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:43.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.276+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:58:43.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.276+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:43.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.276+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:43.283 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:43.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:43.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:43.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:43.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:43.284 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:58:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:58:43.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:43.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:43.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006c950 con 0x7f63d8003840 2021-03-09T10:58:43.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:43.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:43.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e001b610 con 0x7f63d8003840 2021-03-09T10:58:43.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:58:43.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.277+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:43.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.278+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21533 cs=0 l=1 rev1=1 rx=0x7f63e006cae0 tx=0x7f63fc065f20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:43.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.278+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:58:43.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.278+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:43.288 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:43.283+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 67152 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:43.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc04a6d0 con 0x7f63d8003840 2021-03-09T10:58:43.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.284+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:43.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.284+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:43.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.284+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:43.290 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:43.291 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:43.291 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:43.291 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:43.291 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:43.292 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:43.292 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:43.292 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:43.292 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:43.292 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:43.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.285+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02df10 con 0x7f63d8003840 2021-03-09T10:58:43.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.285+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:43.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.285+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:58:43.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.286+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc058a20 con 0x7f63d8003840 2021-03-09T10:58:43.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.840+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:43.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:43.840+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:44.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:44.840+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:44.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:44.840+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:45.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:45.840+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:45.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:45.840+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:45.856 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:45.856+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:46.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:46.840+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:46.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:46.840+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:47.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:47.840+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:47.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:47.840+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:48.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.283+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:58:48.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:48.285 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21533 cs=0 l=1 rev1=1 rx=0x7f63e006cae0 tx=0x7f63fc065f20).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:48.286 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21533 cs=0 l=1 rev1=1 rx=0x7f63e006cae0 tx=0x7f63fc065f20).stop 2021-03-09T10:58:48.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:48.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:48.287 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:48.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=CLOSED pgs=21533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:48.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient: _add_conns ranks=[1,2,0] 2021-03-09T10:58:48.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:48.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:48.288 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00093b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:48.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:48.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:48.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:48.289 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:48.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8003400 2021-03-09T10:58:48.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:48.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e005de80 con 0x7f63d8009130 2021-03-09T10:58:48.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:48.290 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006a260 con 0x7f63e0018390 2021-03-09T10:58:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:48.291 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:48.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00093b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:48.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6406553700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:58:48.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6406553700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:48.292 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.284+0000 7f6406553700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:48.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:48.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:48.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:48.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:48.293 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:48.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:48.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:48.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:48.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 10 monclient: _finish_hunting 0 2021-03-09T10:58:48.294 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 1 monclient: found mon.c 2021-03-09T10:58:48.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:48.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:48.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0079500 con 0x7f63e0018390 2021-03-09T10:58:48.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:48.295 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:48.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006c950 con 0x7f63e0018390 2021-03-09T10:58:48.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 10 monclient: _finish_auth 0 2021-03-09T10:58:48.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.285+0000 7f6406553700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:48.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.286+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00093b0 secure :-1 s=READY pgs=21540 cs=0 l=1 rev1=1 rx=0x7f63e00327c0 tx=0x7f63fc02dff0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:48.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.286+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:58:48.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.286+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:48.298 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:48.294+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 67251 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:48.298 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.294+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc02c860 con 0x7f63e0018390 2021-03-09T10:58:48.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.294+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:48.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.295+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:48.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.295+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:48.299 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:48.299 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:48.300 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:48.300 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:48.300 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:48.300 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:48.300 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:48.301 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:48.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc02d500 con 0x7f63e0018390 2021-03-09T10:58:48.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.295+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:48.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.295+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:58:48.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.296+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63fc029290 con 0x7f63e0018390 2021-03-09T10:58:48.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.840+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:48.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:48.840+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:49.841 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:49.840+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:49.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:49.840+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:50.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:50.840+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:50.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:50.840+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:50.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:50.840+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:50.857 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:50.857+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:51.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:51.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:51.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:51.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:52.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:52.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:52.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:52.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:53.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:53.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00093b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:53.296 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00093b0 secure :-1 s=READY pgs=21540 cs=0 l=1 rev1=1 rx=0x7f63e00327c0 tx=0x7f63fc02dff0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:53.297 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00093b0 secure :-1 s=READY pgs=21540 cs=0 l=1 rev1=1 rx=0x7f63e00327c0 tx=0x7f63fc02dff0).stop 2021-03-09T10:58:53.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:53.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:53.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e00093b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:53.299 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e00093b0 unknown :-1 s=CLOSED pgs=21540 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:53.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:58:53.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:53.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:53.300 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:53.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:53.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:53.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:53.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:53.301 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007b470 con 0x7f63d8003400 2021-03-09T10:58:53.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:53.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:58:53.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:53.302 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00679f0 con 0x7f63d8003840 2021-03-09T10:58:53.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:53.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:53.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:53.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:53.303 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.295+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:53.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:53.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:58:53.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:53.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:53.304 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:53.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:53.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:53.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:53.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:53.305 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:53.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:53.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:53.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:58:53.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:58:53.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:53.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:53.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e006be40 con 0x7f63d8003840 2021-03-09T10:58:53.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:53.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:53.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00666d0 con 0x7f63d8003840 2021-03-09T10:58:53.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:58:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.296+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.297+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21555 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4003c30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.297+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:58:53.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.297+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:53.310 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:53.304+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 67351 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40225e0 con 0x7f63d8003840 2021-03-09T10:58:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.304+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.304+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:53.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.304+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:53.312 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:53.312 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:53.312 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:53.312 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:53.312 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:53.313 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:53.313 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:53.313 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:53.313 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:53.313 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:53.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.304+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4005a00 con 0x7f63d8003840 2021-03-09T10:58:53.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.304+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:53.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.305+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:58:53.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.306+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4005b70 con 0x7f63d8003840 2021-03-09T10:58:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:53.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:53.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:54.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:54.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:54.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:54.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:55.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:55.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:55.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:55.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:55.858 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:55.858+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:58:56.842 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:56.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:56.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:56.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:57.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:57.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:57.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:57.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:58.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.304+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2021-03-09T10:58:58.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.304+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:58:58.306 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.304+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21555 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4003c30).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:58:58.307 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21555 cs=0 l=1 rev1=1 rx=0x7f63f4061070 tx=0x7f63f4003c30).stop 2021-03-09T10:58:58.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:58.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:58:58.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:58:58.308 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=CLOSED pgs=21555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:58:58.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:58:58.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:58.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:58:58.309 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:58.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:58:58.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:58:58.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:58:58.310 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:58.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001cd10 con 0x7f63d8003400 2021-03-09T10:58:58.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:58.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0015df0 con 0x7f63d8009130 2021-03-09T10:58:58.311 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:58:58.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0001470 con 0x7f63e0018390 2021-03-09T10:58:58.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:58:58.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:58:58.312 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:58.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:58:58.313 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.305+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:58:58.314 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.306+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:58:58.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.306+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:58:58.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.306+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:58:58.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.306+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:58:58.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.307+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:58:58.315 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.307+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:58:58.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.307+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:58:58.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:58:58.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:58.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:58.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:58:58.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:58:58.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:58:58.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:58:58.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:58:58.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:58.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e00024a0 con 0x7f63e0018390 2021-03-09T10:58:58.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:58:58.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:58:58.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e006be40 con 0x7f63e0018390 2021-03-09T10:58:58.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:58:58.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:58.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 secure :-1 s=READY pgs=21573 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4034a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:58:58.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:58:58.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.308+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:58:58.321 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:58:58.314+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 67450 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:58:58.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.315+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4035e70 con 0x7f63e0018390 2021-03-09T10:58:58.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.315+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:58:58.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.315+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:58:58.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.315+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:58:58.323 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:58:58.323 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:58:58.324 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:58.324 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:58:58.324 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:58:58.324 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:58:58.324 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:58:58.325 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:58:58.325 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:58:58.325 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:58:58.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.319+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4035e70 con 0x7f63e0018390 2021-03-09T10:58:58.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.319+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:58:58.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.319+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:58:58.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.319+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4046400 con 0x7f63e0018390 2021-03-09T10:58:58.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.842+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:58.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:58.842+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:58:59.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:59.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:58:59.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:58:59.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:00.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:00.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:00.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:00.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:00.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:00.841+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:59:00.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:00.858+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:59:01.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:01.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:01.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:01.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:02.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:02.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:02.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:02.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:03.316 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:59:03.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:03.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 secure :-1 s=READY pgs=21573 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4034a50).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:03.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 secure :-1 s=READY pgs=21573 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4034a50).stop 2021-03-09T10:59:03.317 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:59:03.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:59:03.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e0064b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:03.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e0064b10 unknown :-1 s=CLOSED pgs=21573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:03.318 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,0,1] 2021-03-09T10:59:03.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:03.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:59:03.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:03.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:59:03.319 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:03.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:59:03.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:59:03.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002e520 con 0x7f63d8003400 2021-03-09T10:59:03.320 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:59:03.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63d8009130 2021-03-09T10:59:03.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:59:03.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00679f0 con 0x7f63d8003840 2021-03-09T10:59:03.321 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:59:03.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:59:03.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:03.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:59:03.322 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.314+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:03.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:03.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:59:03.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:59:03.323 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:59:03.325 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:59:03.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:59:03.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:59:03.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:59:03.326 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:03.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:03.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:03.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.315+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:03.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:59:03.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:59:03.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:59:03.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:59:03.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e005e8e0 con 0x7f63d8003840 2021-03-09T10:59:03.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:59:03.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:59:03.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e00024a0 con 0x7f63d8003840 2021-03-09T10:59:03.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:59:03.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:03.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:59:03.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21583 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4009990).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:03.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.316+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:59:03.330 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:59:03.323+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 67551 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:59:03.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f404be90 con 0x7f63d8003840 2021-03-09T10:59:03.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.324+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:59:03.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.324+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:59:03.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.324+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:59:03.332 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:59:03.332 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:59:03.332 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:59:03.333 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:59:03.333 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:59:03.333 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:59:03.333 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:59:03.334 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:59:03.334 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:59:03.334 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:59:03.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.326+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4034e40 con 0x7f63d8003840 2021-03-09T10:59:03.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.326+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:59:03.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.326+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:59:03.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.329+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f400dc70 con 0x7f63d8003840 2021-03-09T10:59:03.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:03.885 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:03.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:04.843 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:04.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:04.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:04.841+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:05.842+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:05.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:05.842+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:05.860 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:05.859+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:59:06.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:06.841+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:06.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:06.842+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:07.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:07.842+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:07.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:07.842+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:08.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:59:08.327 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:08.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21583 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4009990).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:08.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21583 cs=0 l=1 rev1=1 rx=0x7f63f404e010 tx=0x7f63f4009990).stop 2021-03-09T10:59:08.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:59:08.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:59:08.328 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 msgr2=0x7f63e007db90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:08.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=CLOSED pgs=21583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:08.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient: _add_conns ranks=[2,1,0] 2021-03-09T10:59:08.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:08.329 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63e0018390 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:59:08.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:08.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:59:08.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:08.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:59:08.330 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:59:08.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e007a930 con 0x7f63d8003400 2021-03-09T10:59:08.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:59:08.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00145e0 con 0x7f63d8009130 2021-03-09T10:59:08.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:59:08.331 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e004d7b0 con 0x7f63e0018390 2021-03-09T10:59:08.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:59:08.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:59:08.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.324+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:08.332 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.325+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:59:08.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.325+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:08.333 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.325+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:08.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.325+0000 7f6405551700 10 monclient(hunting): get_auth_request con 0x7f63e0018390 auth_method 0 2021-03-09T10:59:08.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.325+0000 7f6405551700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:59:08.334 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.325+0000 7f6405551700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:59:08.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.325+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:59:08.335 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.325+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:59:08.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.325+0000 7f6405551700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:59:08.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:59:08.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:08.336 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:08.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:08.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:08.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 10 monclient: _finish_hunting 0 2021-03-09T10:59:08.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 1 monclient: found mon.c 2021-03-09T10:59:08.337 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:59:08.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:59:08.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e000d840 con 0x7f63e0018390 2021-03-09T10:59:08.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:59:08.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:59:08.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e005e8e0 con 0x7f63e0018390 2021-03-09T10:59:08.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 10 monclient: _finish_auth 0 2021-03-09T10:59:08.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:08.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63e0018390 2021-03-09T10:59:08.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002b450 secure :-1 s=READY pgs=21596 cs=0 l=1 rev1=1 rx=0x7f63f402e010 tx=0x7f63f4009a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:08.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.326+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:59:08.341 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:59:08.334+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 67650 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:59:08.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f401b830 con 0x7f63e0018390 2021-03-09T10:59:08.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.334+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:59:08.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.334+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:59:08.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.334+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:59:08.342 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:59:08.343 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:59:08.343 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:59:08.343 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:59:08.343 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:59:08.343 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:59:08.344 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:59:08.344 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:59:08.344 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:59:08.344 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:59:08.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.334+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f4035e70 con 0x7f63e0018390 2021-03-09T10:59:08.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.334+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:59:08.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.334+0000 7f63ef7fe700 4 set_mon_vals no callback set 2021-03-09T10:59:08.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f4034d80 con 0x7f63e0018390 2021-03-09T10:59:08.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.842+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:08.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:08.842+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:09.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:09.842+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:09.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:09.842+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:10.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:10.842+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:10.842+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:10.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:10.842+0000 7f63ecff9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:59:10.862 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:10.860+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:59:11.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:11.842+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:11.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:11.842+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:12.844 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:12.842+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:12.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:12.842+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:13.338 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.334+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e002b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2021-03-09T10:59:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.334+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e002b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2021-03-09T10:59:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.334+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002b450 secure :-1 s=READY pgs=21596 cs=0 l=1 rev1=1 rx=0x7f63f402e010 tx=0x7f63f4009a70).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2021-03-09T10:59:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.334+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002b450 secure :-1 s=READY pgs=21596 cs=0 l=1 rev1=1 rx=0x7f63f402e010 tx=0x7f63f4009a70).stop 2021-03-09T10:59:13.339 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient: ms_handle_reset current mon [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:59:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient: _reopen_session rank -1 2021-03-09T10:59:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 msgr2=0x7f63e002b450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2021-03-09T10:59:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63e0018390 0x7f63e002b450 unknown :-1 s=CLOSED pgs=21596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2021-03-09T10:59:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient: _add_conns ranks=[0,1,2] 2021-03-09T10:59:13.340 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:13.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient(hunting): picked mon.a con 0x7f63d8003400 addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2021-03-09T10:59:13.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:13.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient(hunting): picked mon.b con 0x7f63d8009130 addr [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2021-03-09T10:59:13.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2021-03-09T10:59:13.341 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient(hunting): picked mon.c con 0x7f63d8003840 addr [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] 2021-03-09T10:59:13.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:59:13.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e001b660 con 0x7f63d8003400 2021-03-09T10:59:13.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:59:13.342 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00266b0 con 0x7f63d8009130 2021-03-09T10:59:13.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient(hunting): start opening mon connection 2021-03-09T10:59:13.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e002edb0 con 0x7f63d8003840 2021-03-09T10:59:13.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f63edffb700 10 monclient(hunting): _renew_subs 2021-03-09T10:59:13.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6406553700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.195:3300/0 2021-03-09T10:59:13.343 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6406553700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:13.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2021-03-09T10:59:13.344 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6405551700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.99:3300/0 2021-03-09T10:59:13.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6405d52700 10 monclient(hunting): get_auth_request con 0x7f63d8003840 auth_method 0 2021-03-09T10:59:13.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6405551700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2021-03-09T10:59:13.345 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6405d52700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2021-03-09T10:59:13.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6405d52700 10 monclient(hunting): _init_auth method 2 2021-03-09T10:59:13.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload 9 2021-03-09T10:59:13.346 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2021-03-09T10:59:13.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.335+0000 7f6405d52700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2021-03-09T10:59:13.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 10 monclient(hunting): handle_auth_done global_id 4208 payload 995 2021-03-09T10:59:13.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 msgr2=0x7f63e000af40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:13.347 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] conn(0x7f63d8009130 0x7f63e000af40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:13.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 msgr2=0x7f63e0064240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2021-03-09T10:59:13.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] conn(0x7f63d8003400 0x7f63e0064240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2021-03-09T10:59:13.348 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 10 monclient: _finish_hunting 0 2021-03-09T10:59:13.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 1 monclient: found mon.c 2021-03-09T10:59:13.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 20 monclient: _un_backoff reopen_interval_multipler now 1 2021-03-09T10:59:13.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:59:13.349 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f63e0010480 con 0x7f63d8003840 2021-03-09T10:59:13.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2021-03-09T10:59:13.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.99:3301/0 2021-03-09T10:59:13.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 1 -- 172.21.15.99:0/2209380211 --> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f63e002d600 con 0x7f63d8003840 2021-03-09T10:59:13.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 10 monclient: _finish_auth 0 2021-03-09T10:59:13.350 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.336+0000 7f6405d52700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:13.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.337+0000 7f6405d52700 1 --2- 172.21.15.99:0/2209380211 >> [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] conn(0x7f63d8003840 0x7f63e007db90 secure :-1 s=READY pgs=21609 cs=0 l=1 rev1=1 rx=0x7f63e001d2b0 tx=0x7f63f004db20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2021-03-09T10:59:13.351 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.337+0000 7f63edffb700 10 client.4208.objecter ms_handle_connect 0x7f63d8003840 2021-03-09T10:59:13.352 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.337+0000 7f63edffb700 10 client.4208.objecter resend_mon_ops 2021-03-09T10:59:13.352 INFO:tasks.ceph.mon.c.smithi099.stderr:2021-03-09T10:59:13.344+0000 7f7c225dd700 -1 mon.c@2(probing) e1 get_health_metrics reporting 67736 slow ops, oldest is log(1 entries from seq 8 at 2021-03-09T08:48:11.319758+0000) 2021-03-09T10:59:13.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.344+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63d8003840 2021-03-09T10:59:13.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.344+0000 7f63edffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2021-03-09T10:59:13.353 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.344+0000 7f63edffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2021-03-09T10:59:13.354 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.344+0000 7f63edffb700 10 monclient: dump: 2021-03-09T10:59:13.354 INFO:teuthology.orchestra.run.smithi099.stderr:epoch 1 2021-03-09T10:59:13.354 INFO:teuthology.orchestra.run.smithi099.stderr:fsid e7417d55-b73a-46ac-8f0e-ad6e583f612d 2021-03-09T10:59:13.354 INFO:teuthology.orchestra.run.smithi099.stderr:last_changed 2021-03-09T08:47:25.175756+0000 2021-03-09T10:59:13.355 INFO:teuthology.orchestra.run.smithi099.stderr:created 2021-03-09T08:47:25.175756+0000 2021-03-09T10:59:13.355 INFO:teuthology.orchestra.run.smithi099.stderr:min_mon_release 17 (quincy) 2021-03-09T10:59:13.355 INFO:teuthology.orchestra.run.smithi099.stderr:election_strategy: 3 2021-03-09T10:59:13.355 INFO:teuthology.orchestra.run.smithi099.stderr:0: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.a 2021-03-09T10:59:13.355 INFO:teuthology.orchestra.run.smithi099.stderr:1: [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon.b 2021-03-09T10:59:13.356 INFO:teuthology.orchestra.run.smithi099.stderr:2: [v2:172.21.15.99:3301/0,v1:172.21.15.99:6790/0] mon.c 2021-03-09T10:59:13.356 INFO:teuthology.orchestra.run.smithi099.stderr: 2021-03-09T10:59:13.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.345+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f00282a0 con 0x7f63d8003840 2021-03-09T10:59:13.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.345+0000 7f63edffb700 10 monclient: handle_config config(0 keys) v1 2021-03-09T10:59:13.356 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.345+0000 7f63effff700 4 set_mon_vals no callback set 2021-03-09T10:59:13.357 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.346+0000 7f63edffb700 1 -- 172.21.15.99:0/2209380211 <== mon.2 v2:172.21.15.99:3301/0 3 ==== mgrmap(e 4) v1 ==== 70293+0+0 (secure 0 0 0) 0x7f63f0041d30 con 0x7f63d8003840 2021-03-09T10:59:13.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.843+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:13.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:13.843+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:14.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:14.843+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:14.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:14.843+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:15.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:15.843+0000 7f63ecff9700 10 monclient: tick 2021-03-09T10:59:15.845 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:15.843+0000 7f63ecff9700 20 monclient: _check_auth_rotating not needed by client.admin 2021-03-09T10:59:15.863 INFO:teuthology.orchestra.run.smithi099.stderr:2021-03-09T10:59:15.861+0000 7f63ee7fc700 10 client.4208.objecter tick 2021-03-09T10:59:16.023 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-09T10:59:16.025 DEBUG:teuthology.task.console_log:Killing console logger for smithi099 2021-03-09T10:59:16.027 DEBUG:teuthology.task.console_log:Killing console logger for smithi195 2021-03-09T10:59:16.027 DEBUG:teuthology.exit:Finished running handlers